SlowGuess/NAACL_2021_OCR huggingface dataset:
Ramit Sawhney*
Shivam Agarwal
Manipal Institute of Technology
shivamag99@gmail.com
It is challenging to design profitable and practical trading strategies, as stock price movements are highly stochastic, and the market is heavily influenced by chaotic data across sources like news and social media. Existing NLP approaches largely treat stock prediction as a classification or regression problem and are not optimized to make profitable investment decisions. Further, they do not model the temporal dynamics of large volumes of diversely influential text to which the market responds quickly. Building on these shortcomings, we propose a deep reinforcement learning approach that makes time-aware decisions to trade stocks while optimizing profit using textual data. Our method outperforms state-of-the-art in terms of risk-adjusted returns in trading simulations on two benchmarks: Tweets (English) and financial news (Chinese) pertaining to two major indexes and four global stock markets. Through extensive experiments and studies, we build the case for our method as a tool for quantitative trading.
The stock market, a financial ecosystem involving quantitative trading and investing, observed a market capitalization exceeding \(US60 trillion as of the year 2019. Stock trading presents lucrative opportunities for investors to utilize the market as a platform for investing funds and maximizing profits. However, making profitable investment decisions is challenging due to the market's volatile, noisy, and chaotic nature (Tsay, 2005; Adam et al., 2016). Research at the intersection of Natural Language Processing (NLP) and finance presents encouraging prospects in stock prediction (Jiang, 2020). Conventional work forecasts future trends by modeling numerical historical stock data (Lu
Arnav Wadhwa*
MIDAS, IIIT Delhi
arnavw96@gmail.com
Figure 1: Here, we show how tweets about Tesla and Moderna influence investors' opinions and impact the stocks over a day and the upcoming week. The tweets by the Tesla CEO Elon Musk lead to massive price drops in Tesla's stock, and Moderna's positive news attracts investments in its stock. A profitable trading decision would entail selling off Tesla's shares (if already held) and buying Moderna's stock in such a scenario.
et al., 2009; Bao et al., 2017). However, price signals alone can not capture market surprises, mergers, acquisitions, and company announcements. Such events, often reported across financial news and social media, have strong influence over market dynamics (Laakkonen, 2004). For instance, prices immediately react to breaking news about the related company (Busse and Green, 2002). Such reactions conform to the Efficient Market Hypothesis (EMH), a hypothesis in finance which states that financial markets are informationally efficient and prices reflect all available market information at any given time (Malkiel, 1989).
The abundance of stock affecting information across news and social media online inspires the adoption of natural language processing to study the interplay between textual data and stock prices (Oliveira et al., 2017; Xu and Cohen, 2018). However, unlike numerical data, the study of natural language is more challenging. Individual tweets or news headlines may not be informative enough, and analyzing them together can provide a greater
Association for Computational Linguistics: Human Language Technologies, pages 4018-4030
context, as shown in Figure 1. Moreover, the timing of their release plays a critical role as stock markets rapidly react to new information (Foucault et al., 2016). Furthermore, not each news story or tweet holds the potential to influence stock trends as texts have a diverse influence on prices (Hu et al., 2017). These observations suggest benefits in factoring in the time-aware dependence and diverse influence of text while analyzing natural language.
Despite profitability being the prime objective of quantitative trading, existing natural language processing methods for stock prediction (Hu et al., 2017; Xu and Cohen, 2018; Du and Tanaka-Ishii, 2020) are commonly formulated as classification or regression tasks, and are not directly optimized towards profit generation. Such methods face fundamental drawbacks. First, they do not innately incorporate the decision making and strategies involved in quantitative trading, in turn limiting potential profitability. Second, they have limited practical applicability as they do not factor in the monetary resources available and financial assets (stocks) held with a trader at each trading time step. This gap presents a new research direction where profit generation can be directly optimized by modeling the complex sequential decision-making process in quantitative trading as a Reinforcement Learning (RL) task. Owing to its nature, RL formulation is directly suitable to the problem of quantitative trading as it provides the potential to automatically learn the adjustment of investment budgets across stocks in portfolios while taking into account the configuration of investments made in the past.
Contributions: We formulate stock prediction as a reinforcement learning problem (\$3) and present PROFIT: Policy for Return Optimization using FInancial news and online Text, a deep reinforcement learning approach that leverages financial news and tweets to model stock-affecting signals and optimize trading decisions for increasing profitability. PROFIT accounts for the monetary resources available and the existing portfolio to execute profitable trades at any given time. Through extensive experiments (\$5) on English and Chinese text corresponding to the NASDAQ, Shanghai, Shenzhen, and Hong Kong markets, we show that PROFIT outperforms state-of-the-art methods in terms of risk adjusted returns by over  $13\%$  and minimizes extreme losses by over  $16\%$  (\$6.1,  $\S 6.2$ ). Using exploratory analyses (\$6.3), we show PROFIT's practical and real-world applicability.
Reinforcement Learning and Natural Language Processing Lately, reinforcement learning has influenced solutions for a wide variety of natural language processing tasks and applications. These include, but are not limited to information extraction (Qin et al., 2018), social media analysis (Zhou and Wang, 2018), text classification (Wu et al., 2018a), extractive (Narayan et al., 2018) and abstractive (Chen and Bansal, 2018) text summarization, neural machine translation (Wu et al., 2018b), text-based games (He et al., 2016a; Ammanabrolu and Riedl, 2019), knowledge-based question answering (Hua et al., 2020) and much more. For these tasks and applications, deep reinforcement learning methods have been more successful in modeling the complexities involved in natural language, such as the processing of large vocabularies and phrases that otherwise make action selection (He et al., 2016a,b) arduous for RL methods that do not exploit deep networks as function approximators. However, most existing methods for a variety of tasks face a fundamental drawback – they do not take into account the influence of the inherent dynamic temporal irregularities and the variably influential nature of text while modeling a time-series of language data over action selection and sequential decision making.
Reinforcement Learning in Finance Recent years have witnessed the adoption of reinforcement learning in the financial realm to solve tasks such as portfolio management (Filos, 2019; Almahdi and Yang, 2019), equity asset reallocation (Meng and Khushi, 2019; Katongo and Bhattacharyya, 2021), cryptocurrency trading (Jiang et al., 2017; Jiang and Liang, 2017; Lucarelli and Borrotti, 2019; Ye et al., 2020) and much more. Existing work heavily relies on factors such as technical indicators (Wang et al., 2019; Liu et al., 2020) to model price signals, or use simple numeric features like sentiment scores from text (Yang et al., 2018) to model stock affecting information reflected across news items. However, these methods experience two significant drawbacks. Firstly, despite their success, the performance of such methods depend largely on the quality of external feature representations (for instance, sentence embeddings (Ye et al., 2020)) of text. Secondly, methods that only use prices exhibit lower practical applicability to real-world trading, owing to the lack of information in prices alone.
We formulate stock trading as a reinforcement learning problem. Let  $S = \{s_1, s_2, \ldots, s_N\}$  denote a set of  $N$  stocks. We aim to design a trading agent that learns to interact with the stock market environment by leveraging stock-aflicting signals present across financial news items and tweets to trade stocks. In the context of an agent, an interaction comprises observing the environment state at any particular time-step to generate an action, and reach the next time-step to receive a reward along with the next state. The typical Markov Decision Process (MDP) description is widely adopted for RL tasks where environments are fully-observable. However, in the stock market, prices are influenced by numerous macro- and micro-economic factors, investor opinions about stocks formed through social media, financial news, and countless other sources. Thus, it becomes pragmatically and computationally impractical to observe and incorporate stock affecting information from all possible sources to make trading decisions. As the stock markets and the underlying factors that drive stock prices are not fully-observable, Partially Observable MDP (POMDP) provides a natural generalization of the MDP to model the stock trading environment (Jaakkola et al., 1995). Hence, the key components of the stock trading environment considered and developed in this study are as follows:
State observations: At a time-step  $\tau$ , the state  $s_{\tau}$  comprises a trading-account observation  $o_{\tau}$  and a market-information observation  $o_m$ . The trading-account observation  $o_{\tau}$  comprises the account balance and the number of shares owned corresponding to each stock at time-step  $\tau$ . The market-information observation  $o_m$  comprises stock-relevant news or tweets released during a T-day lookback period (days  $\in [\tau - T + 1, \tau]$ ). The text input in  $o_m$  is structured such that it comprises all stock relevant text in a lookback window of length  $T$  in a hierarchical fashion within and across days. The orders made through the trading actions taken by the reinforcement learning agent would have minute impacts on the overall market trends, thus having little to no direct influence on the market-information observations.
Trading actions: The agent can buy, sell, or hold the shares for each stock at the time-step  $\tau$ . We compute a vector of actions  $a_{\tau}$  over the set of stocks  $S$  as decisions made by the agent, which result in
an increase, decrease, or no change in the number of stocks shares  $h$ . One of three possible actions is taken on each stock  $s$ :
- Buying  $k[s] \in [1, h[s]]$  shares results in  $h_{\tau + 1}[s] = h_{\tau}[s] + k[s]$ , where  $k[s] \in \mathbb{Z}^+$ .
- Holding  $k[s] \in [1, h[s]]$  shares results in  $h_{\tau + 1}[s] = h_{\tau}[s]$ .
- Selling  $k[s] \in [1, h[s]]$  shares lead to  $h_{\tau + 1}[s] = h_{\tau}[s] - k[s]$ .
Note that the trading actions at time-step  $\tau$  directly impact the trading-account observation at time-step  $\tau + 1$ ,  $o_{\tau + 1}$ .
Rewards: We define the reward as the change in the value when an action is taken at state  $s_{\tau}$  to arrive at new state  $s_{\tau + 1}$ . Corresponding to each state change, we define a return  $r$ , as:
$$
r \left(s _ {\tau}, a _ {\tau}, s _ {\tau + 1}\right) = \left(b _ {\tau + 1} + p _ {\tau + 1} ^ {T} h _ {\tau + 1}\right) - \left(b _ {\tau} + p _ {\tau} ^ {T} h _ {\tau}\right) - c _ {\tau} \tag {1}
$$
where  $b_{\tau}$  is the account balance,  $p_{\tau}$  is a vector that represents the stock prices,  $h_{\tau}$  denotes the stock shares in the trading account, and  $c_{\tau}$  denotes the transaction costs incurred at time-step  $\tau$ . To maximize the earned profit, we aim to design a reinforcement learning agent that maximizes the cumulative change  $r(s_{\tau}, a_{\tau}, s_{\tau + 1})$ .
4 Proposed Approach: PROFIT
We adopt reinforcement learning to optimize profitability in quantitative trading. To this end, we introduce PROFIT, a deep reinforcement learning approach for text-based stock trading, as shown in Figure 2. For this study, we make use of a custom policy network that hierarchically and attentively learns time-aware representations of news and tweets to trade stocks. In practice, PROFIT's proposed policy network is generalizable across various actor-critic reinforcement learning methods that exploit neural networks as function approximators. Moreover, PROFIT is compatible with any custom policy network of the same nature that can handle textual time-series data.
4.1 Deep Reinforcement Learning
We base PROFIT on the Deep Deterministic Policy Gradient (DDPG) framework (Lillicrap et al., 2015), which bridges the gap between policy gradient (Sutton et al., 2000) and value approximation methods (Watkins and Dayan, 1992) for RL. The
Figure 2: PROFIT: Trading policy network (top), deep reinforcement learning for stock trading (bottom).
DDPG decouples the trading action selection and the trading action evaluation processes into two separate jointly learned networks: the actor network, and the critic network. The actor-network  $\mu$ , parameterized by  $\theta$ , takes the observations at state  $s_{\tau}$  as input, and outputs the trading actions  $a_{\tau}$ . The critic-network  $Q$ , parameterized by  $\phi$ , takes the observations at state  $s_{\tau}$  and trading actions  $a_{\tau}$  from the actor as input. It then outputs a scalar  $Q(s_{\tau}, a_{\tau})$  to evaluate the action  $a_{\tau}$ .
For each state  $s_{\tau}$ , the agent performs an action  $a_{\tau}$ , receives a reward  $r_{\tau}$ , and reaches the next state  $s_{\tau + 1}$ . These transitions represented as  $(s_{\tau}, a_{\tau}, s_{\tau + 1}, r_{\tau})$  are stored in a replay buffer  $\mathcal{D}$ . Subsequently, a mini-batch  $B$  comprising  $N$  transitions is sampled from  $\mathcal{D}$  for updating the model. For each batch  $B$ , PROFIT minimizes the following loss  $L$  with respect to  $\phi$  to update the critic as:
$$
y _ {\tau} = r _ {\tau} + \gamma Q ^ {\phi^ {\prime}} (s _ {\tau + 1}, \mu^ {\theta^ {\prime}} (s _ {\tau + 1})), \qquad (2)
$$
$$
L = \mathbb {E} \left[ \left(y _ {\tau} - Q ^ {\phi} \left(s _ {\tau}, a _ {\tau}\right)\right) ^ {2} \right] \tag {3}
$$
where  $y_{\tau}$  is the updated Q-value,  $\gamma$  is a discount factor,  $\theta$  and  $\theta^{\prime}$ ,  $\phi$  and  $\phi^{\prime}$  are the two copy parameters of the policy  $\mu$  and the value function  $Q$ , respectively. The actor is updated using the policy gradient  $\nabla_{\theta}J$  via backpropagation through time as:
$$
\nabla_ {\theta} J = \mathbb {E} \left[ \nabla_ {a} Q ^ {\phi} \left(s _ {\tau}, \mu^ {\theta} \left(s _ {\tau}\right)\right) \nabla_ {\theta} \mu^ {\theta} \left(s _ {\tau}\right) \right] \quad (4)
$$
In the above equations,  $\theta$  and  $\theta^{\prime}$ ,  $\phi$  and  $\phi^{\prime}$  are the two copy parameters of the policy  $\mu$  and the value function  $Q$ , respectively. For a detailed explanation of the framework, we refer the readers to Lillicrap
et al. (2015). Next, we define the trading policy network, which takes the observations at state  $s_{\tau}$  as input to generate stock trading actions  $a_{\tau}$ . We use the same architecture for defining the actor and the critic networks.
4.2 Trading Policy Network
To generate trading actions, we first learn representations for each stock  $s \in S$  using the T-day market-information observation  $o_{m}$ , and the trading-account observation  $o_{\tau}$  at the time-step  $\tau$ . For this study, we derive inspiration from Hu et al. (2017); Sawhney et al. (2020, 2021) to design the policy network. However, it is important to note that PROFIT is compatible with any general deep network that is capable of handling time-series of textual data. We specifically adopt the following network as it inherently covers a breadth of components that are proved beneficial for designing language-based systems for stock trading.
First, PROFIT's policy encodes the texts  $t$  corresponding to a stock  $s$  released in a day using BERT (Devlin et al., 2019). We tokenize and truncate the input text (t) for each news item or tweet and feed it to BERT. We then aggregate the final hidden states (the final-layer transformer outputs) of the input to get the encoded representation (m, size 768) as as  $m = \mathrm{BERT}(t) \in \mathbb{R}^d$ ,  $d = 768$ . We also experiment with the [CLS] token and other pooling techniques such as maximum of hidden states and concatenation of mean and maximum of hidden states but do not obtain better results.
For each stock  $s$  on a day  $i$ , a variable number (K) of tweets (t) are posted at irregular times (k). LSTMs though able to capture the sequential con
text dependencies in text over time, assume inputs to be equally spaced in time. However, the intervals between release of consecutive news items or tweets can vary widely, from a few seconds to many hours, and that can have a drastic impact on their influence on the market (O'Hara, 2015). Thus, we use a time-aware LSTM (TLSTM) (Baytas et al., 2017), to capture the irregularities in the release of text, and encode them for a stock  $s$  on a day  $i$ .
All news and tweets in a day might not be equally informative, and may have diverse influence over a stock's trend (Barber and Odean, 2007). We use an intra-day attention mechanism (Qin et al., 2017) that allows the trading agent to emphasize texts likely to have a more substantial impact on price. The attention mechanism learns to adaptively aggregate the variable number of hidden states of the t-LSTM into an intra-day text information vector. We combine these representations across days in a hierarchical fashion using an LSTM.
We use attention again over the outputs of the LSTM to obtain a market-information vector  $p_{\tau}$  comprising financial signals across tweets or news items released over the lookback. Lastly, we concatenate the trading-account observation  $o_{\tau}$  at state  $s_{\tau}$ , with the market-information vector  $p_{\tau}$  to form an overall stock-level representation  $z_{\tau} = [o_{\tau}, p_{\tau}]$ .
Trading actions: We concatenate the stock-representations  $z_{\tau}$  to form a feature vector  $Z$  across stocks for day  $\tau$ . We then feed  $Z$  to a feed-forward network, followed by a tanh activation function, which outputs actions  $a_{\tau}$  to buy, hold or sell the shares of each stock  $s \in S$  at the time-step  $\tau$ .
5.1 Datasets and Stock Markets
US S&P 500<sup>1</sup> (Xu and Cohen, 2018): Comprises 109,915 English tweets from the social media platform Twitter spanning January 2014 to December 2015, related to 88 high-trade-volume stocks from the NASDAQ Exchange forming the S&P 500 index. NASDAQ is a fairly volatile (Schwert, 2002) US exchange. The stocks are categorized into 9 industries:<sup>2</sup> Basic Materials, Consumer Goods, Healthcare, Services, Utilities, Conglomerates, Financial, Industrial Goods and Technology. Xu and Cohen (2018) extracted stock specific tweets using
regex queries made of stock ticker symbols, for instance, $AMZN for Amazon, where $ acts as a cashtag on the platform Twitter).
China & Hong Kong $^{3}$  (Huang et al., 2018): Comprises 90, 361 financial news headlines in Chinese. The headlines span January 2015 to December 2015, and are originally aggregated by Wind $^{4}$  from major financial website like Sina $^{5}$  and Hexun. $^{6}$  The news headlines are related to 85 China A-share stocks from the Shanghai, Shenzhen and the Hong Kong Stock Exchanges. Huang et al. (2018) extracted news from major financial websites covering corporate news across Mainland China and Hong Kong.
Pre-processing: We pre-process English tweets using the NLTK $^{7}$  (Twitter mode), for treatment of URLs, identifiers (@) and hashtags (#). We adopt the Bert-Tokenizer for tokenization of tweets. For the English tweets, we use the pre-trained BERT-base-cased $^{8}$  model. For the Chinese news, we adopt the Chinese-BERT-base $^{8}$  model, having 12 layers and 110M parameters. We use character-based tokenization for the Chinese headlines. We collect prices from Yahoo Finance. $^{9}$  We align trading days by dropping data samples that do not possess tweets for a consecutive 7-day window, and further align the data across windows for stocks to ensure that data is available for all days in the window for the same set of stocks. We split the US S&P 500 dataset temporally based on date ranges from January 01, 2014 to July 31, 2015 for training, August 01, 2015 to September 30, 2015 for validation, and October 01, 2015 to January 01, 2016 for testing. We split the China & Hong Kong dataset temporally based on date ranges from January 01, 2015 to August 31, 2015 for training, September 01, 2015 to September 30, 2015 for validation, and October 01, 2015 to January 01, 2016 for testing all models and experiments.
5.2 PROFIT Training Setup
We conduct all experiments on a Tesla P100 GPU. We use grid search to find optimal hyperparameters based on the validation Sharpe Ratio (§5.3) for all
<sup>3</sup>China & Hong Kong dataset: https://pan.baidu.com/s/1mhCLJJi
4https://www.wind.com.cn/en/wft.html
5http://finance.sina.com.cn/
http://www.hexun.com/
<sup>7</sup>https://www.nltk.org/
$^{8}$ www.github.com/google-research/bert
9Prices from: https://finance.yahoo.com/
$^{1}$ US S&P 500 dataset: www.github.com/yumoxu/stocknet-dataset
<sup>2</sup>https://finance.yahoo.com/industries
models. We build the RL agent in Python programming language using PyTorch and employ OpenAI gym to implement the stock trading environment. We explore the length of the lookback period  $T \in \text{range}[2, 10]$  days. Across both the datasets, we obtain that the model best performs for a week-long lookback – i.e. 7 days. We explore the hidden state dimension for both TLSTM and LSTM  $d \in [32, 64, 128]$  (we achieve the best performance for:  $d = 64$ , both for the TLSTM and the LSTM) across both the datasets. We factor the time elapsed between the successive posting of texts at the common finest granularity available across the datasets – i.e. 1 minute intervals. We use the Xavier initialization (Glorot and Bengio, 2010) to initialize all network weights. We use an exponential learning rate scheduler (Li and Arora, 2019) with a decay rate of 0.001 and an initial learning rate of  $7e - 5$ . For each dataset, we train PROFIT using the Adam optimizer (Kingma and Ba, 2014).