SlowGuess/NAACL_2021_OCR huggingface dataset:
To assess the profitability and trading performance of all methods, we compute the Sharpe ratio (SR), its variant Sortino Ratio (StR), the Cumulative Return (CR), and the Maximum Drawdown (MDD). The Sharpe Ratio is a measure of the return of a portfolio compared to its risk (Sharpe, 1994). We calculate SR by computing the earned return  $R_{a}$  in excess of the risk-free return $^{10}$ $R_{f}$ , defined as:  $\mathrm{SR} = \frac{E[R_a - R_f]}{std[R_a - R_f]}$ . The Sortino Ratio is a variation of the Sharpe Ratio, which uses an asset's standard deviation of negative portfolio returns (downside deviation,  $\sigma_d$ ) as:  $\mathrm{StR} = \frac{E[R_a - R_f]}{\sigma_d}$ . The StR is a useful way to evaluate an investment's return for a given level of bad risk, and provides a better view of the risk-adjusted return – as positive volatility is essentially considered beneficial. The CR is the change in the investment over time and is computed using the initial  $(b_0)$  and the final  $(b_f)$  account balance as:  $\mathrm{CR} = \frac{b_f - b_0}{b_0} * 100$ . The MDD measures the maximum loss from a peak  $r_p$  to a trough  $r_t$  of a portfolio, and is defined as:  $\mathrm{MDD} = \frac{r_t - r_p}{r_p} * 100$ . Larger values (in magnitude) of MDD indicate higher volatility. MDD is an indicator used to assess the relative riskiness of one stock trading strategy versus another, as it focuses on capital preservation, which is a key concern for most investors. For instance, two trading strategies may have the same volatility, average outper
formance, and tracking error, but their maximum drawdowns compared to the benchmark can differ drastically. Investors typically prefer the strategy with lower maximum drawdowns.
5.4 Practical Trading Constrains
The following assumptions and constraints reflect concerns for practical stock trading. PROFIT accounts for various elements of the trading process and the financial aspects like transaction costs, market liquidity, and risk-aversion (Yang et al., 2020).
Non-negative account balance: Ideally, the allowed trading actions should not result in a negative account balance. Based on the stock-level actions generated at time  $\tau$ , the stocks are divided into sets for selling, buying, and holding, non-overlapping sets. The constraint for non-negative balance is that for any given time step  $\tau$ , the sum of account balance  $b_{\tau}$ ; the money gained through selling the stocks in set S; and the money spent for acquiring the stocks in the buying set: should be positive, or at minimum zero.
Transaction costs: For each trade, various types of transaction costs such as exchange fees, execution fees, and SEC fees are incurred. Further, in practice, different brokers have different commission fees, and despite these variations, we assume our transaction costs to be  $0.1\%$  of the value of each trade (either buy or sell).
5.5 Baseline Approaches
We compare PROFIT with baselines spanning different formulations: regression, classification, ranking, and reinforcement learning. We follow the same preprocessing protocols as proposed by the original works and adopt their implementations, if available publicly.
Regression (REG) These methods regress return ratios from past data and trade the top stocks.
- W-LSTM: LSTMs with stacked autoencoders that encode noise-free data obtained through wavelet transform of prices (Bao et al., 2017).
AZFinText: Proper noun-based text representations fed to Support Vector Regression for forecasting returns (Schumaker and Chen, 2009).
Classification (CLF) The following methods classify movements as [up, down, neutral] and trade the stocks where prices are expected to rise.
10T-Bill rates: https://home.treasury.gov/
Formulation	US S&P 500	China & Hong Kong
\(CR\uparrow\)	\(SR\uparrow\)	\(StR\uparrow\)	\(MDD\downarrow\)	\(CR\uparrow\)	\(SR\uparrow\)	\(StR\uparrow\)	\(MDD\downarrow\)
Regression	9.62 ± 2.16	0.76 ± 0.21	0.99 ± 0.40	18.98 ± 4.56	24.81 ± 11.56	1.02 ± 0.29	1.49 ± 0.37	14.34 ± 5.63
Classification	10.06 ± 2.73	0.89 ± 0.26	1.15 ± 0.51	19.06 ± 5.07	25.72 ± 13.29	1.03 ± 0.22	1.56 ± 0.43	15.88 ± 5.58
Ranking (Sawhney et al., 2021)	21.45 ± 6.78	0.95 ± 0.11	1.35 ± 0.27	16.93 ± 5.58	33.25 ± 15.12	1.19 ± 0.16	1.67 ± 0.34	10.45 ± 2.36
Reinforcement Learning	29.64 ± 8.22	1.03 ± 0.24	1.87 ± 0.65	5.01 ± 4.21	40.88 ± 13.04	1.29 ± 0.32	1.99 ± 0.61	6.78 ± 6.09
Table 1: Trading performance over different problem formulations (mean of 5 runs). All formulations use the same base architecture defined in PROFIT's policy network to model stock affecting text over the lookback period.
- TSLDA: Topic Sentiment Latent Dirichlet Allocation, a generative model jointly exploiting topics and sentiments in textual data (Nguyen and Shirai, 2015).
- StockEmb: Stock embeddings acquired using prices, and dual vector (word-level vectors and context-level vectors) representation of texts (Du and Tanaka-Ishii, 2020).
- SN - HFA: StockNet - HedgeFundAnalyst, a variational autoencoder with attention on texts and prices (Xu and Cohen, 2018).
- MAN-SF (text only): BERT based hierarchical encoder for financial text using hierarchical temporal attention (Sawhney et al., 2020).
- Chaotic: A Hierarchical Attention Network using GRU encoders with temporal attention applied on text within days, and the days in the lookback period (Hu et al., 2017).
Ranking (RAN) The following methods rank stocks to select most profitable trading candidates.
- R-LSTM: Utilizes 5-day, 10-day, 20-day, and 30-day averages and closing prices of stocks to train an LSTM model (Feng et al., 2019).
- RankNet: A DNN that utilizes sentiment-based shock and trend scores to optimize a probabilistic ranking function (Song et al., 2017).
Reinforcement Learning (RL) The following approaches optimize quantitative trading through reinforcement learning.
- iRDPG: An imitative Recurrent Deterministic Policy Gradient (RDPG) algorithm exploiting temporal stock price features, while optimizing the Sharpe Ratio as the reward (Liu et al., 2020).
- AlphaStock: An LSTM based network to model prices, comprising attention to model inter-stock cross relations (Wang et al., 2019).
- S-Reward: Inverse reinforcement learning method to model relations between sentiments and returns (Yang et al., 2018).
- SARL: A Deterministic Policy Gradient with augmented states, comprising stock prices and encoded news (Ye et al., 2020).
6.1 Stock Trading Problem Formulation
We experiment with four different formulations for neural stock trading in Table 1. For each formulation, we treat our custom policy trading network as the base architecture for modeling stock affecting textual information over the lookback period. We find that classification and regression formulations generate relatively low profits compared to others. This is likely as trades in such methods are not optimized for the overall profit as a reward. Moreover, another limitation of classification and regression approaches is that the trading strategy needs to be defined manually. Next, we find that reinforcement learning provides the best performance as it allows PROFIT to enjoy a more granular control over trading actions and learn to optimize the strategy directly for making profitable trades using text. Further, we also observe that trading under RL formulation experiences the lowest MDD, likely as the agent has more flexibility in selecting the trades, which leads to lower losses. Next, we study how different baseline stock trading networks across the four formulations perform compared to PROFIT.
6.2 Performance Comparison with Baselines
We now compare PROFIT's profitability (Sharpe Ratio) and risk in investment (Maximum Drawdown) against baseline approaches in Table 2. PROFIT generates higher risk-adjusted returns and experiences lower losses than all methods, as we show in Figure 3. We find methods that incorporate stock affecting information from textual sources generate profits higher or comparable to price-only methods. These results indicate that textual sources
US S&P 500	China & Hong Kong
Models & Components	SR↑	MDD↓	SR↑	MDD↓
REG	W-LSTM (Bao et al., 2017)	P	0.41 ± 0.15	32.91 ± 7.91	0.49 ± 0.13
AZFinText (Schumaker and Chen, 2009)	T + P	0.40 ± 0.10	31.46 ± 5.91	0.50 ± 0.09
CLF	TSLDA (Nguyen and Shirai, 2015)	T + P	0.39 ± 0.08	31.72 ± 6.71	0.51 ± 0.12
StockEmb (Du and Tanaka-Ishii, 2020)	T + P + A	0.51 ± 0.14	22.01 ± 10.87	0.74 ± 0.21
SN - HFA (Xu and Cohen, 2018)	T + P + A	0.81 ± 0.08	12.15 ± 2.01	0.93 ± 0.09
MAN-SF (Text only) (Sawhney et al., 2020)	T + A	0.80 ± 0.11	18.09 ± 7.24	1.01 ± 0.15
Chaotic (Hu et al., 2017)	T + A	0.86 ± 0.21	15.49 ± 5.38	0.95 ± 0.37
RAN	R-LSTM (Feng et al., 2019)	P	0.78 ± 0.19	21.42 ± 3.21	0.96 ± 0.05
RankNet (Song et al., 2017)	T	0.87 ± 0.09	10.40 ± 2.90	0.95 ± 0.10
RL	iRDPG (Liu et al., 2020)	P	0.79 ± 0.14	17.71 ± 9.56	1.03 ± 0.28
AlphaStock (Wang et al., 2019)	P + A	0.71 ± 0.24	11.54 ± 6.91	0.95 ± 0.24
S-Reward (Yang et al., 2018)	T	0.73 ± 0.16	10.46 ± 7.22	1.08 ± 0.39
SARL (Ye et al., 2020)	T + P	0.91 ± 0.13	8.38 ± 4.95	1.10 ± 0.19
PROFIT (Ours)	T + A	1.03 ± 0.24	5.01 ± 4.21	1.29 ± 0.32
Table 2: Profitability comparison against baseline approaches (mean of 5 runs) (\$5.5). Within Components,  $\mathrm{T} =$  Text,  $\mathrm{P} =$  Prices,  $\mathrm{A} =$  Attention across modalities. Green and blue depict best and second-best results, respectively.
can augment neural stock prediction, as they potentially help capture classic financial anomalies such as the over- and under-reaction of asset prices to news (Bondt and Thaler, 1985; Corgnet et al., 2013). This observation also follows prior research that shows financial text are generally better indicators of market volatility, compared to price signals (Atkins et al., 2018). In general, we observe that ranking and reinforcement learning methods generate high returns as they are directly optimized towards profit generation. Further, reinforcement learning approaches are typically more profitable as the trading agents optimize every trading action for profit generation directly, unlike ranking, where the task is only to select profitable stocks to trade. These observations validate the premise of formulating quantitative trading as a reinforcement learning problem, compared to conventionally adopted regression and classification formulations.
Despite the 2015-16 Chinese Market Turbulence Recession $^{11}$  (Liu et al., 2016), the lower MDD of PROFIT indicates the trading agent's ability to respond to bearish markets $^{12}$ , and its performance is attributable to the following reasons. Amongst competitive baselines, PROFIT's policy design differentiates it from others, as it captures the hierarchical dependencies in the news and attentively learns to emphasize crucial trading indicators during such turbulent economies. The attention mechanisms potentially account for financial phenomena such as the calendar (Jacobs and Levy, 1988) and
Figure 3: Capital growth (initial $100,000) through PROFIT's trades compared against baseline methods.
the day-of-the-week (Halil, 2001) effects, and better distinguish noise inducing text from relevant market signals to minimize false evaluations and overreactions (De Long et al., 1989). Further, Jiao et al. (2020) show that frequent news media coverage is an indicator of a decrease in stock volatility. Through its time-aware mechanism, the agent can incorporate such frequencies and learn to trade less volatile stocks to execute low-risk and high-profit trades even in bearish market scenarios.
6.3 Parameter Analysis: Probing Sensitivity
Lookback period length T Here, we study how PROFIT's performance varies with the length of lookback period  $T \in [2, 10]$  days in Figure 4. Lower performance indicates the inability of shorter lookbacks to capture stock affecting market information, as public information requires time to absorb into price movements (Luss and D'Aspremont, 2015). As we increase  $T$ , we observe a deterioration in
<sup>11</sup>https://www.vox.com/2015/7/8/8908765/chinas-stock-market-crash-explained  
<sup>12</sup>Bearish markets are those that experience prolonged price declines, experience high volatility and risk on investments.
Figure 4: Sensitivity to parameters T and  $b_{0}$
the trading performance. This indicates that larger lookbacks allow the inclusion of stale information from older days having relatively lower influence on prices (Bernhaedt and Miao, 2004). We observe optimal performance for mid-sized lookbacks.
Initial trading balance  $\mathbf{b}_0$  To further analyze PROFIT's trading performance, we simulate the cumulative returns for different initial trading amounts. Financial studies highlight that larger investments are prone to higher risk (Stout, 1995), as higher budgets allow increased risk-taking abilities. Ghysels et al. (2005) find significantly positive relations between larger risk and higher returns (risk-return tradeoff).<sup>13</sup> PROFIT's performance is akin to this phenomena as we observe generally high rewards even for riskier decisions taken on larger investments, as shown in Figure 4. We attribute PROFIT's versatility to its policy design that allows diverse trading choices based on resource availability. These results indicate that PROFIT holds practical applicability to investors across diverse economic milieus: from individual traders to larger firms having greater investment margins.
We propose PROFIT, a deep RL approach for quantitative trading using textual data across online news and tweets. To model the market information, PROFIT hierarchically learns temporally relevant signals from texts in a time-aware fashion, and directly optimizes trading actions towards profit generation. Through extensive analyses on English tweets and Chinese news spanning four markets, we highlight PROFIT's real-world applicability. In trading simulations on the S&P 500 and China A-shares indexes, PROFIT outperforms baselines in terms of profitability and risk in investment.
There is an ethical imperative implicit in this growing influence of automation in market behavior, and it is worthy of serious study (Hurlburt et al., 2009; Cooper et al., 2020). Since financial markets are transparent (Bloomfield and O'Hara, 1999), and heavily regulated (Edwards, 1996), we discuss the ethical considerations pertaining to our work. Following (Cooper et al., 2016), we emphasize on three ethical criteria for automated trading systems and discuss PROFIT's design with respect to these criteria.
Prudent System A prudent system "demands adherence to processes that reliably produce strategies with desirable characteristics such as minimizing risk, and generating revenue in excess of its costs over a period acceptable to its investors" (Longstreth, 1986). PROFIT is directly optimized towards profit-generation and minimizing investor risk by selectively investing in the less volatile stocks ( $\S 6.2$ ), and generates risk-adjusted returns: Sharpe Ratio, as shown in Table 2.
Blocking Price Discovery A trading system should not block price discovery and not interfere with the ability of other market participants to add to their own information (Angel and McCabe, 2013). For example, placing an extremely large volume of orders to block competitor's messages (Quote Stuffing) or intentionally trading with itself to create the illusion of market activity (Wash Trading). PROFIT does not block price discovery in any form.
Circumventing Price Discovery A trading system should not hide information, such as by participating in dark pools or placing hidden orders (Zhu, 2014). We evaluate PROFIT only on public data in highly regulated stock markets.
Despite these considerations, it is possible for PROFIT, just as any other automated trading system, to be exploited to hinder market fairness. We follow broad ethical guidelines to design and evaluate PROFIT, and encourage readers to follow both regulatory and ethical considerations pertaining to the stock market.
Klaus Adam, Albert Marcet, and Juan Pablo Nicoli. 2016. Stock market volatility and learning. The Journal of Finance, 71(1):33-82.
<sup>13</sup>https://www.investopedia.com/terms/r/riskreturntradeoff.asp
Saud Almahdi and Steve Y Yang. 2019. A constrained portfolio trading system using particle swarm algorithm and recurrent reinforcement learning. Expert Systems with Applications, 130:145-156.
Prithviraj Ammanabrolu and Mark Riedl. 2019. Playing text-adventure games with graph-based deep reinforcement learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3557-3565, Minneapolis, Minnesota. Association for Computational Linguistics.
James J Angel and Douglas McCabe. 2013. Fairness in financial markets: The case of high frequency trading. Journal of Business Ethics, 112(4):585-595.
Adam Atkins, Mahesan Niranjan, and Enrico Gerding. 2018. Financial news predicts stock market volatility better than close price. The Journal of Finance and Data Science, 4(2):120-137.
Wei Bao, Jun Yue, and Yulei Rao. 2017. A deep learning framework for financial time series using stacked autoencoders and long-short term memory. PLOS ONE.
Brad M. Barber and Terrance Odean. 2007. All That Glitters: The Effect of Attention and News on the Buying Behavior of Individual and Institutional Investors. The Review of Financial Studies, 21(2):785-818.
Inci M. Baytas, Cao Xiao, Xi Zhang, Fei Wang, Anil K. Jain, and Jiayu Zhou. 2017. Patient subtyping via time-aware LSTM networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '17, page 65-74, New York, NY, USA. Association for Computing Machinery.
Dan Bernhaedt and Jianjun Miao. 2004. Informed trading when information becomes stale. The Journal of Finance, 59(1):339-390.
Robert Bloomfield and Maureen O'Hara. 1999. Market transparency: who wins and who loses? The Review of Financial Studies, 12(1):5-35.
Werner F. M. De Bondt and Richard Thaler. 1985. Does the stock market overreact? The Journal of Finance, 40(3):793-805.
Jeffrey A. Busse and T. Green. 2002. Market efficiency in real time. Journal of Financial Economics, 65(3):415-437.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting.
Ricky Cooper, Michael Davis, Andrew Kumiega, and Ben Van Vliet. 2020. Ethics for automated financial markets. Handbook on Ethics in Finance, pages 1-18.
Ricky Cooper, Michael Davis, Ben Van Vliet, et al. 2016. The mysterious ethics of high-frequency trading. Business Ethics Quarterly, 26(1):1-22.
Brice Corgnet, Praveen Kujal, and David Porter. 2013. Reaction to public information in markets: how much does ambiguity matter? The Economic Journal, 123(569):699-737.
J. BRADFORD De Long, ANDREI SHLEIFER, LAWRENCE H. SUMMERS, and ROBERT J. WALDMANN. 1989. The size and incidence of the losses from noise trading. The Journal of Finance, 44(3):681-696.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Xin Du and Kumiko Tanaka-Ishii. 2020. Stock embeddings acquired from news articles and price history, and an application to portfolio optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3353-3363, Online. Association for Computational Linguistics.
Franklin R Edwards. 1996. The new finance: regulation and financial stability. American Enterprise Institute.
Fuli Feng, Xiangnan He, Xiang Wang, Cheng Luo, Yiqun Liu, and Tat-Seng Chua. 2019. Temporal relational ranking for stock prediction. ACM Transactions on Information Systems, 37(2):1-30.
Angelos Filos. 2019. Reinforcement learning for portfolio management. arXiv preprint arXiv:1909.09571.
Thierry Foucault, Johan Hombert, and Ioanid Rosu. 2016. News trading and speed. The Journal of Finance, 71(1):335-382.
Eric Ghysels, Pedro Santa-Clara, and Rossen Valkanov. 2005. There is a risk-return trade-off after all. Journal of Financial Economics, 76(3):509 - 548.
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256.
Berument Hakan Kiymaz Halil. 2001. The day of the week effect on stock market volatility. Journal of economics and finance, 25(2):181-193.
Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Li-hong Li, Li Deng, and Mari Ostendorf. 2016a. Deep reinforcement learning with a natural language action space. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1621-1630, Berlin, Germany. Association for Computational Linguistics.
Ji He, Mari Ostendorf, Xiaodong He, Jianshu Chen, Jianfeng Gao, Lihong Li, and Li Deng. 2016b. Deep reinforcement learning with a combinatorial action space for predicting popular Reddit threads. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1838-1848, Austin, Texas. Association for Computational Linguistics.
Ziniu Hu, Weiqing Liu, Jiang Bian, Xuanzhe Liu, and Tie-Yan Liu. 2017. Listening to chaotic whispers: A deep learning framework for news-oriented stock trend prediction. In Proceedings of the 11 international conference on web search and data mining.
Yuncheng Hua, Yuan-Fang Li, Gholamreza Haffari, Guilin Qi, and Tongtong Wu. 2020. Few-shot complex knowledge base question answering via meta reinforcement learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5827-5837, Online. Association for Computational Linguistics.
Jieyun Huang, Yunjia Zhang, Jialai Zhang, and Xi Zhang. 2018. A tensor-based sub-mode coordinate algorithm for stock prediction.
George F Hurlburt, Keith W Miller, and Jeffrey M Voas. 2009. An ethical analysis of automation, risk, and the financial crises of 2008. IT professional, 11(1):14-19.
Tommi Jaakkola, Satinder P Singh, and Michael I Jordan. 1995. Reinforcement learning algorithm for partially observable markov decision problems. In Advances in neural information processing systems, pages 345-352.
Bruce I Jacobs and Kenneth N Levy. 1988. Calendar anomalies: Abnormal returns at calendar turning points. Financial Analysts Journal, 44(6):28-39.
Weiwei Jiang. 2020. Applications of deep learning in stock market prediction: recent progress.
Zhengyao Jiang and Jinjun Liang. 2017. Cryptocurrency portfolio management with deep reinforcement learning. In 2017 Intelligent Systems Conference (IntelliSys), pages 905-913. IEEE.
Zhengyao Jiang, Dixing Xu, and Jinjun Liang. 2017. A deep reinforcement learning framework for the financial portfolio management problem. arXiv preprint arXiv:1706.10059.