SlowGuess/ACL_2020_OCR huggingface dataset:
Xin Du
Department of Advanced Interdisciplinary
Studies, Graduate School of Engineering,
duxin@cl.rcast.u-tokyo.ac.jp
Kumiko Tanaka-Ishii
Research Center of
Advanced Science and Technology,
kumiko@cl.rcast.u-tokyo.ac.jp
Previous works that integrated news articles to better process stock prices used a variety of neural networks to predict price movements. The textual and price information were both encoded in the neural network, and it is therefore difficult to apply this approach in situations other than the original framework of the notoriously hard problem of price prediction. In contrast, this paper presents a method to encode the influence of news articles through a vector representation of stocks called a stock embedding. The stock embedding is acquired with a deep learning framework using both news articles and price history. Because the embedding takes the operational form of a vector, it is applicable to other financial problems besides price prediction. As one example application, we show the results of portfolio optimization using Reuters & Bloomberg headlines, producing a capital gain 2.8 times larger than that obtained with a baseline method using only stock price data. This suggests that the proposed stock embedding can leverage textual financial semantics to solve financial prediction problems.
News articles influence the dynamics of financial markets. For example, after the release of breaking news, the share prices of related stocks are often observed to move. This suggests the possibility of using natural language processing (NLP) to aid traders by analyzing this influence between news article texts and prices.
Recent studies (Ding et al., 2015; Hu et al., 2018; Chen et al., 2019; Yang et al., 2018) have indeed reported that news articles can be leveraged to improve the accuracy of predicting stock price movements. These previous works have used deep learning techniques. They train neural networks with article texts and financial market prices, attempting
to improve price prediction. In these approaches, the overall mutual effect between texts and prices is distributed over the neural network, which makes it difficult to extract this effect and apply it to tasks other than price prediction.
Therefore, we take a new approach by explicitly describing this mutual effect in terms of a vector. A stock is represented by a vector so that its inner product with an embedding of a text produces a larger value when the text is more related to the stock. In the rest of the paper, we call this vector a stock embedding.
The names of stocks, such as "AAPL" (the ticker symbol for Apple Inc.), typically appear in a financial news article text. Because these names form part of the text, usual NLP techniques can be applied to acquire an embedding of a stock. Such general textual embedding, however, does not incorporate the financial reality of stock price changes. Hence, the proposed stock embedding represents the price as well as the semantics of the text, as we acquire it by training on both news articles and stock prices. Precisely, our stock embedding is trained through a binary classification problem, namely, whether a stock price goes up or down in comparison with the previous day's price. As a result, an acquired stock embedding captures the relation between a stock name and a news article even when the article has no direct mention of the stock. Our stock embedding can be considered as one technique to specialize, or ground, a symbol that has a practical reality outside of text.
Furthermore, two major advantages come with the vector form of our stock embedding. The first is that the training can be effectuated for all stocks at once, rather than stock by stock. This is an important advantage to alleviate data sparseness and prevent overfitting, as discussed in Section 4.
The second advantage lies in the portability of a vector. In contrast to previous works, in which
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3353-3363
stock-specific information was distributed among the parameters of a neural network, a vector representing all the characteristics of a stock is much easier to extract and apply to other uses besides price prediction.
Hence, this paper shows an example of portfolio optimization, one of the most important applications in finance. To the best of our knowledge, this is the first report of incorporating NLP into modern portfolio theory (Markowitz, 1952). Our method differs from previous works that used NLP to enhance investment strategies. Many previous works focused on stock price forecasting only (Ding et al., 2015; Hu et al., 2018) and did not attempt to apply the learned results to other financial tasks. Another previous work (Song et al., 2017) investigated portfolios with texts. It obtained a ranking of stocks from texts by using a neural network technique and then evaluated investment in the highest/lowest ranked stocks. That work was not based on modern portfolio theory, however, nor did it integrate price and text data. In contrast, our method uses NLP in addition to price data to acquire a general representation in the form of an embedding applicable to different targets. In our experiments, a portfolio generated using stock embeddings achieved an annual gain 2.8 times greater than that of a portfolio generated with price data only. This provides evidence that the stock embedding well encodes both text and price information.
The main idea of this article is based on important techniques of NLP. It is now common to represent discrete entities in natural language by continuous vectors. These vectors are called "embeddings" and usually obtained from neural network models. Examples include the word embedding (Mikolov et al., 2013), phrase embedding (Zhang et al., 2014), sentence embedding (Lin et al., 2017), and event embedding (Ding et al., 2016).
One advantage of these continuous representations is that the geometry of an embedding system contains rich semantic information, as has been discovered at many levels (Mikolov et al., 2013; Reif et al., 2019). The acquisition of stock embeddings in this paper is based on the original idea developed for linguistic entities. Here, we extend the idea further so that the embeddings reflect the reality of a stock market outside text.
A stock embedding is trained using the attention
mechanism (Bahdanau et al., 2015), which is another current NLP technique. The basic idea of the original attention mechanism is to assign higher weights to more relevant word vectors and make the weights adaptive to different contexts.
Our framework is based on the classification task for text-driven stock price movement, which has been studied intensely as follows. Early research on exploiting financial news articles for better stock price prediction dates back to Ou and Penman (1989), in which financial indicators were extracted manually from financial statements. Later, in Fung et al. (2002), NLP methods were adopted for automatic text feature extraction. Since the 2000s, Twitter and other text-centered social media platforms have become essential sources of financial signals. Bollen et al. (2011) found evidence for causality between the public mood extracted from tweets and the Dow Jones Industrial Average index. In Nguyen et al. (2015), post texts collected from the Yahoo! Finance Message Board were used to predict whether the prices of 18 US stocks would rise or drop on the next trading day.
As deep learning methods for NLP have become more common, many papers have reported the use of neural networks for text-driven stock classification (or prediction) tasks. Ding et al. (2015) proposed an event embedding to represent a news headline with a vector and used a convolutional neural network for classification. In that work, all the event embeddings of news articles published on the same day were simply averaged to summarize that day's market information.
Hu et al. (2018) was among the first works that applied the attention mechanism to the task of news-driven stock price movement classification. They developed a dual-level attention framework, in which news articles were assigned different weights depending on the output of a logistic regression component with a bias term, so that the most informative news articles were "highlighted." The method of weighting news articles in this paper is similar to that previous work. The stock-specific information in Hu et al. (2018) was encoded in the neural network, however, making it focused on the price prediction task. In contrast, we represent such stock-specific information by the stock embedding, i.e., a vector, which is easy to interpret geometrically and extract for other applications.
For one such application, we evaluated our stock embedding in terms of portfolio optimization. To
the best of our knowledge, this is the first paper applying NLP techniques to modern portfolio theory. We use the mean-variance minimization portfolio model (introduced in Section 7) proposed in Markowitz (1952), which directly led to the capital asset pricing model (Sharpe, 1964).
3 News-Driven Stock Price Classification
In this paper, the stock embedding is trained with a deep learning system through binary classification of price movements. Let  $p_t$  be the stock price on day  $t$ , and let  $y_t$  be the desired output of the system. Here,  $t \in \{1, 2, \dots, T\}$ , and  $T$  is the number of trading days in the considered time period. The binary classification problem indicates that  $y_t$  is classified in the following way:
$$
y _ {t} = \left\{ \begin{array}{l l} 1, & p _ {t} \geq p _ {t - 1} \\ 0, & p _ {t} <   p _ {t - 1}. \end{array} \right. \tag {1}
$$
To train such a deep learning system, news articles are used as the input. In this work, news articles are considered daily (i.e., treated in units of days). We denote the set of articles published on day  $t$  by  $N_{t}$ , and each article by  $n_i \in N_t$ , with  $i = 1, \ldots, |N_t|$ . This paper considers a time window around day  $t$ , denoted as  $[t - d_1, t + d_2]$  given two constants  $d_1, d_2$ . Let  $N_{[t - d_1, t + d_2]}$  be the set of news articles published within the time window.
When  $d_{2} = -1$ , indicating the use of articles until day  $t - 1$ , the task is called prediction, as the training does not use any articles published on or after day  $t$ . In general, this task is acknowledged as very hard (Fama, 1970; Basu, 1977; Timmermann and Granger, 2004) according to the efficient-market hypothesis (EMH)<sup>1</sup>, and such prediction provides only a limited gain, if any. Note that previous NLP studies concerning stock prices were all aimed at this hard problem (Ding et al., 2015; Hu et al., 2018; Xu and Cohen, 2018; Yang et al., 2018).
On the other hand, when  $d_{2} \geq 0$ , this paper refers to the task as classification. The performance on classification shows how well the model understands a news article. Because the prediction problem is too hard and offers limited gain, as proven by many previous works, our target lies in classification. The aims are thus to acquire embeddings that are highly sensitive to textual context and to
apply them to tasks other than price prediction. Therefore, in this paper, we set  $d_{1} = 4$  and  $d_{2} = 0$ .
Let the classification model be represented by a mapping  $f$ . The probability that the price of a stock  $j$ , where  $j = 1, \dots, J$ , goes up on day  $t$  is
$$
\hat {y} _ {t} ^ {j} = f \left(N _ {[ t - 4, t ]}\right). \tag {2}
$$
In the process of model optimization, the model should reduce the mean cross-entropy loss between every true label  $y_{t}^{j}$  and its corresponding estimate  $\hat{y}_{t}^{j}$ , as follows:
$$
l ^ {j} = - \frac {1}{T} \sum_ {t = 1} ^ {T} \left(y _ {t} ^ {j} \log \hat {y} _ {t} ^ {j} + (1 - y _ {t} ^ {j}) \log (1 - \hat {y} _ {t} ^ {j})\right).
$$
This function describes the loss for only one stock, but a stock market includes multiple stocks. This work considers all stocks in a market equally important. The overall loss function is therefore a simple average of the cross-entropy loss for all stocks, i.e.,  $l = \left( \sum_{j=1}^{J} l^j \right) / J$ .
4 Method to Acquire Stock Embeddings
Let  $s_j$  represent a stock embedding, where  $j = 1,2,\dots,J$ . This is initialized as a random vector and then trained via a neural model to obtain  $s_j$ , whose inner product with the embedding of a related text becomes large. This section describes the proposed method to acquire stock embeddings by building up a neural network for price movement classification.
The neural network consists of two parts: a text feature distiller and a price movement classifier.
Text feature distiller. The text feature distiller first converts every news article  $n_i$  into a pair of vectors  $(n_i^K, n_i^V)$  corresponding to "key" and "value" vectors, respectively. Let  $N_t^K = \{n_i^K\}_t$ ,  $N_t^V = \{n_i^V\}_t$  denote the sets of key/value vectors of the articles released on day  $t$ . Such dual-vector representation of a text was proposed and adopted successfully in Miller et al. (2016) and Daniluk et al. (2017). The pair of vectors contains the semantic information of the article text at two different levels. Roughly,  $n_i^K$  represents the article at the word level, whereas  $n_i^V$  represents it at the context level.
The text feature distiller calculates the attention score for every article  $i$  published on day  $t$ . The attention score between article  $i$  and stock  $j$  is given by the inner product of the two vectors  $n_i^K$  and  $s_j$ :
$$
\mathrm {s c o r e} _ {i, j} = n _ {i} ^ {K} \cdot s _ {j}.
$$
Note that there are other possible definitions of this inner product, such as the cosine similarity or
<sup>1</sup>According to the EMH, in an "efficient" market, prices reflect the true values of assets by having incorporated all past information, so nobody can predict the price. The EMH is hypothesized to hold but has also attracted criticism.
a generalized inner product using some arbitrary function. Because this work focuses on the most basic capability of the stock embedding, it uses the most basic inner product (i.e., the dot product).
Let  $\alpha_{i}^{j}$  denote the weight put on news article  $i$  with respect to stock  $j$ , to classify whether the stock price will go up or down. With the use of score $_{i,j}$  defined above,  $\alpha_{i}^{j}$  is given as the following:
$$
\alpha_ {i} ^ {j} \equiv \frac {\exp (\mathrm {s c o r e} _ {i , j})}{\sum_ {i ^ {\prime}} \exp (\mathrm {s c o r e} _ {i ^ {\prime} , j})}.
$$
$\alpha_{i}^{j}$  is thus acquired as the softmax function of the scores across the articles released on the same day.
By using  $\alpha_{i}^{j}$  as the weights put on news articles, we compute the market status of stock  $j$  on day  $t$  as the following, which is the input to the classifier:
$$
m _ {t} ^ {j} = \sum_ {n _ {i} ^ {V} \in N _ {t} ^ {V}} \alpha_ {i} ^ {j} n _ {i} ^ {V}. \tag {3}
$$
Therefore,  $m_t^j$  is computed over a set of  $n_i^V$ , representing the context of texts on day  $t$ . We call  $m_t^j$  the market vector, to which we will return in Section 6.
Price movement classifier. The input of the price movement classifier is a sequence of vectors,  $M_{[t - 4,t]}^j = [m_{t - 4}^j,m_{t - 3}^j,\dots ,m_t^j]$ , with respect to stock  $j$ . This is processed by a recurrent neural network using a bidirectional gated recurrent unit (Bi-GRU). The choice of a Bi-GRU was made by considering the model capacity and training difficulty. The classifier estimates the probability  $\hat{y}_t^j$ :
$$
\begin{array}{r l} h _ {t} ^ {O} & = \operatorname {G R U} \left(M _ {[ t - 4, t ]} ^ {j}\right), \\ \hat {y} _ {t} ^ {j} & = \sigma (\operatorname {M L P} \left(h _ {t} ^ {O}\right)), \end{array} \tag {4}
$$
where  $\sigma (x) = 1 / (1 + \exp (-x))$ , and GRU and MLP stand for the Bi-GRU and a multilayer perceptron, respectively. An optional re-weighting technique over the GRU's output vectors  $h_{\tau}^{O}(\tau \in [t - 4,t])$  (Hu et al., 2018) can be applied. In this case, after the first line of formula (4), the re-weighting is conducted in the following way:
$$
{h ^ {O}} {= \sum_ {\tau = t - 4} ^ {t} \beta_ {\tau} h _ {\tau} ^ {O},}
$$
and this  $h^O$  becomes the input of the second line instead of  $h_t^O$ . Here,  $\beta_{\tau}$ , the weight for day  $\tau$ , decides how much one day is considered in the classification. In our implementation,
$$
\beta_ {\tau} = \frac {\exp (v _ {\tau - t} \cdot h _ {\tau} ^ {O})}{\sum_ {\xi = - 4} ^ {0} \exp (v _ {\xi} \cdot h _ {t + \xi} ^ {O})},
$$
where the vector  $v_{\xi}$  differentiates the temporal effects of news articles released around day  $t$ .  $v_{\xi}$
Figure 1: Illustration of the classifier sharing mechanism across stocks on day  $t$ : (a) one independent classifier per stock, and (b) a shared classifier across stocks.  $|N_{t}|$  denotes the number of news articles on day  $t$ .
is initialized randomly and trained via the neural network. See Hu et al. (2018) for the details.
Such formulation of neural network training has the advantage of avoiding overfitting. A common problem in the task of stock movement classification or prediction is small sample sizes, especially when adopting units of days. In contrast, the proposed model does not suffer from small sample sizes, because the price movement classifier can be trained across all the stocks by sharing one classifier, rather than by generating one classifier for each individual stock like in many previous works (Ding et al., 2015; Hu et al., 2018; Xu and Cohen, 2018). We call this a classifier sharing mechanism.
Figure 1 illustrates the difference between models with and without classifier sharing. The upper figure (a) shows the conventional setting without sharing, in which  $J$  classifiers are generated, one for each stock. In contrast, the lower figure (b) shows one classifier generated for all stocks. This setting enables learning of the correlation among stocks, in addition to avoiding overfitting and the problem of small sample sizes. Specifically, the classifier is shared across all stocks, thus achieving a sample size about 50 to 100 times larger.
5 Dataset and Settings to Acquire Stock Embeddings
We used two news article datasets to build stock embeddings: the Wall Street Journal (WSJ, in the
Dataset	Period	# of articles	# of days / trading days	Mean # of articles per day (std)	Mean # of words per article (std)
Wall Street Journal (WSJ)	2000/01-2015/12	403, 207	5,649/4,008	71.4 (41.1)	28.5 (9.0)
Reuters & Bloomberg (R&B)	2006/10-2013/11	551, 479	2,605/1,794	211.7 (257.1)	9.34 (1.73)
Table 1: Basic information on the two news article datasets.
following) dataset and the Reuters & Bloomberg (R&B) dataset $^2$ , as listed in Table 1. WSJ contains around 400,000 news headlines published across 16 years, whereas R&B contains around 550,000 articles across 7 years. Compared with R&B, WSJ has a relatively more uniform distribution of news articles across time (see the standard deviations listed in parentheses in the fifth column of Table 1). Following previous studies reporting that the main body of a news text produces irrelevant noise (Ding et al., 2015), we extracted only the headlines in both datasets.
As for the stocks, we selected two subsets of the stocks in Standard & Poor's S&P 500 index, one for each of the WSJ and R&B datasets. These subsets consisted only of stocks that were mentioned in no fewer than 100 different news articles, so that mutual effects between the articles and the price history would appear pretty often in the texts. More importantly, this ensured that keyword retrieval-based methods that locate related articles by explicit keyword matching could be applied for comparison. For the WSJ and R&B datasets, the subsets had 89 and 50 stocks, respectively. All other stocks were removed from consideration.
As seen in formula (2), the input for the neural network is  $N_{[t - 4,t]}$ , the set of articles around day  $t$  and the output is  $y_{t}^{j}$ . The label  $y_{t}^{j}$  is the binarized price movement of stock  $j$  at day  $t$ . This is measured by the log-return between two subsequent days:
$$
\log \operatorname {r e t u r n} _ {t} ^ {j} = \log p _ {t} ^ {j} - \log p _ {t - 1} ^ {j}.
$$
The distribution of log-returns is typically bell shaped with a center close to 0, as also mentioned in Hu et al. (2018). The return values of the days were separated into three categories of "negative," "ambiguous," and "positive" by the use of thresholds<sup>3</sup>. Here, "ambiguous" refers to those samples close to 0.0, which were removed. Thus, by using only the clearly negative and positive days, the returns were binarized.
Through such filtering, the number of samples for each stock became about two-thirds of the number of all trading days, or around  $^{4}$  2600 and 1200 samples for each stock, for the WSJ and R&B datasets, respectively.
5.2 Deep Learner System Settings
The Adam optimizer (Kingma and Ba, 2015) was used with cosine annealing (Loshchilov and Hutter, 2017) to train the neural network. The initial learning rate was set to 5e-4. The mini-batch size was 64. We stopped the training process when the value of the loss function with respect to the validation set no longer dropped, and then we measured the accuracy on the test set for evaluation.
As for the dual-vector representation of news article texts, introduced in Section 4, the key and value vectors were calculated as described here. The key vector  $n_i^K$  is defined as follows by using word embeddings  $w_k$  acquired by Word2vec:
$$
n _ {i} ^ {K} = \frac {\sum_ {k} \gamma_ {k} w _ {k}}{\sum_ {k} \gamma_ {k}},
$$
where  $\gamma_{k} = \mathrm{TF}_{k}\cdot \mathrm{IDF}_{k}$  is the TFIDF (Manning and Schütze, 2001) score of word  $k$ . The dimension of  $n_i^K$  equals that of the Word2vec model trained on the news corpus, i.e., 64 in our implementation.
As for the value vector  $n_i^V$ , we used vectors acquired through a BERT encoder. We used the pretrained BERT model available from Google Research, with 24 layers trained on an uncased corpus. This model outputs vectors of 1024 dimensions, but we reduced the dimensions to 256 by using principal component analysis (PCA), to suppress the number of parameters in the neural network. Along with the effect of the stock embedding, the effect of the dual-vector representation (DVR) is also
4The number of samples after filtering differed slightly among stocks, because the distribution of log-returns differed, while the same thresholds were used.
We chose this method after examining several options, including the smooth inverse frequency (SIF) (Arora et al., 2017), TFIDF-weighted word embeddings, and several other methods. We found that TFIDF-weighted word embeddings with Word2vec worked best.
6BERT (Bidirectional Encoder Representations from Transformer) is a neural network model (Devlin et al., 2019) that can be used to encode text into vectors with a fixed dimension.
This dataset was made open source in Ding et al. (2015).
<sup>3</sup>We used the thresholds  $[-0.0053, 0.0079]$  for the WSJ dataset and  $[-0.0059, 0.0068]$  for the R&B dataset. The margins were asymmetric around 0 because these datasets had slightly more "rising" days than "declining" ones.
evaluated in the following section.
6 Effect of Stock Embedding on Price Movement Classification
The basic effect of the stock embedding was evaluated through the performance on the price movement classification task, as stated in Section 3.
The whole dataset described in Section 5.1 was randomly divided into nonoverlapping training/validation/test sets in the ratios of  $0.6 / 0.2 / 0.2$ . The training/validation/test parts did not share any samples from the same dates. Every method below was tested for 10 different random divisions, and the average performance is reported here.