SlowGuess/ACL_2020_OCR huggingface dataset:
The proposed model is abbreviated as  $WA + CS + DVR$ , for weighted average with classifier sharing and dual-vector representation. For an ablation test, four models were considered, which varied the market vector of the day (defined in formula (3) in Section 4 (Ding et al., 2015)) and were with or without the dual-vector representation and classifier sharing (Ding et al., 2015; Hu et al., 2018; Xu and Cohen, 2018; Yang et al., 2018), as follows.
Simple average: The simple average of the text representations of the same day is taken as the market vector of the day, as proposed by Ding et al. (2015).
Weighted average (WA): As stated in formula (3), the market vector of the day is averaged by using the weights from the stock-text inner products, as proposed in Hu et al. (2018). Note again that their work did not apply classifier sharing but instead produced one classifier for each stock, nor did it adopt the dual-vector representation.
WA + classifier sharing (CS): This refers to WA with classifier sharing across stocks. This variant does not adopt the dual-vector representation, i.e.,  $n_i^K$  is set equal to  $n_i^V$  for every news article  $i$ . Thus, the same BERT text embedding is used for both  $n_i^K$  and  $n_i^V$ .
WA + dual-vector representation (DVR): This refers to WA with the dual-vector representation of news texts. This variant does not adopt classifier sharing.
Furthermore, to examine the effect of the data size, we tested different dataset portions: 1 year, 3 years, and the whole dataset. Therefore, the experimental variants involved five methods (four
comparison + our proposal) and three data sizes, or a total of 15 experiments.
Figure 2 summarizes the complete experimental results. The uppermost bar of each bar group, in red, corresponds to our model with classifier sharing (CS) and the dual-vector representation (DVR). The other bars, in orange, blue, purple, and green, correspond to the four ablation variants. The ablation datasets with only 1-year data contained around 150 training samples and were too small for most variants to work well, yet our proposed model,  $WA + CS + DVR$ , could still obtain positive results (classification accuracy over  $50\%$ ). With the 3-year datasets, our  $WA + CS + DVR$  model widened the performance gap, whereas the simple average and weighted average models still failed to work better than random guessing. These results show the superiority of our model in handling the overfitting problem with small datasets.
Finally, the significant differences between  $WA + CS + DVR$  (in red) and  $WA + CS$  (in blue) and between  $WA + DVR$  (in orange) and  $WA$  (in purple) strongly supported the advantage of adopting the dual-vector representation (DVR), especially when classifier sharing was combined.
Overall, our model successfully achieved  $68.8\%$  accuracy for the R&B dataset, which was significantly better than any of the other four variants.
7 Portfolio Optimization
Thus far, the evaluation on classification has shown the capability of our framework in understanding news articles. For financial applications, however, the task must be in the form of prediction; that is, it must produce some gain ahead of the time when a news article is published. As one such predictive example, we present portfolio optimization, one of the most important financial tasks, and we show how our stock embedding can be applied to it.
A portfolio is essentially a set of weights assigned to stocks, representing the proportions of capital invested in them. Intuitively, a portfolio bears a bigger risk if a large proportion is invested in two highly positively correlated stocks, rather than two uncorrelated or negatively correlated stocks. Based on this idea, the mean-variance minimization model in Markowitz (1952) is formulated as follows:
$$
\min  _ {w} \quad \text {r i s k} = w ^ {T} \Sigma w \tag {5a}
$$
$$
\text {s u b j e c t} \quad w ^ {T} r = E, \tag {5b}
$$
Figure 2: Mean classification accuracy percentages (with SD in parentheses) over 10 replications.
$$
w ^ {T} \mathbf {1} = 1, \tag {5c}
$$
$$
0 \leq w _ {j} \leq 1 \quad j = 1, \dots , J, \tag {5d}
$$
where  $\Sigma$  is the risk matrix;  $w$  is the vector of investment weights;  $r$  is a vector such that  $r_j$  equals the mean historic return of stock  $j$ ; 1 is a vector of ones; and  $E$ , the expected portfolio return, is a parameter decided by an investor's preference. Note that higher  $E$  usually means higher risk born by the investor.
In the original model of Markowitz,  $\Sigma$  is the covariance matrix of the historic return time series of stocks,  $\Sigma_{ij} = \mathrm{Cov}(\{r_i\}_t,\{r_j\}_t)$ $(i,j\in$ $\{1,\dots ,J\})$  . According to Markowitz (1952), the solution of this optimization problem, which can be obtained via quadratic programming, gives the portfolio with the smallest risk for an expected overall return  $E$
Using the covariance matrix as the risk matrix  $\Sigma$  is limited, however, for two reasons. First, the overwhelming noise in price movements prevents accurate estimation of the covariance. More importantly, it ignores the events described in news articles that indeed cause price movements.
On the other hand, the stock embeddings built here provide much abundant textual information for defining  $\Sigma$ . Concretely,
$$
\Sigma_ {i, j} = \cos \left(s _ {i}, s _ {j}\right).
$$
This should work because the stock embedding reflects a stock's responsiveness to a certain class of news events. In other words, close stock embeddings indicate a correlated response pattern to an event described in news articles. Stock embeddings capture this correlation much better than the covariance matrix does, and this correlation is what a good portfolio relies on.
By solving the same optimization problem but with a different matrix  $\Sigma$ , we get another vector of
investment ratios,  $w$ , with respect to the stocks. By virtually investing according to  $w$  and observing the result within a certain period,  $\Sigma$  can be evaluated. For each of the WSJ and R&B datasets, we ran one investment simulation for various definitions of  $\Sigma$ , as follows.
S&P 500 index: As a market baseline, we used an S&P 500 index portfolio, in which all 505 stocks in the index were considered and the investment weight  $w_{j}$  was in proportion to the market capitalization of stock  $j$ . The price history of the portfolio was provided by Dow Jones. This method did not use  $\Sigma$  to form the portfolio.
S&P 89*/50*: This approach was the same as above but with the set of stocks reduced to those tested in our work, as explained in Section 5.1: 89 stocks for the WSJ dataset<sup>7</sup>, and 50 for the R&B dataset.
Covariance matrix of historic stock returns:
$\Sigma$  was the covariance matrix as originally proposed by Markowitz.
Word2vec-general: (text only)  $\Sigma$  was the cosine matrix of the word embeddings trained on general corpora (fastText word embeddings (Bojanowski et al., 2017) were used in our experiments). For each stock, we used the word embedding of its ticker symbol, e.g., the word embedding of "AAPL" for Apple Inc.
Word2vec-news: (text only)  $\Sigma$  was the cosine matrix of the word embedding vectors trained
<sup>7</sup>The S&P 89* portfolio was evaluated during the period of 2001 to 2016. The market capitalization history of the stocks before the year 2005 is not available, so the record was estimated for this missing period. First, the number of shares outstanding was extrapolated from the data of 2005-2016, in which the values were pretty stable during the whole period. The market capitalization was then acquired by multiplying the price by the shares outstanding.
on news text corpora. We used the full text of the R&B dataset for training, in which all mentions of a stock in the text were replaced by the stock's ticker symbol.
Covariance  $\cdot$  stock embedding: (text and price)  $\Sigma$  was the result of element-wise multiplication of the covariance matrix and the cosine matrix of the stock embeddings.
Weighted BERT: (text only)  $\Sigma$  was the cosine matrix of stock vectors acquired as follows, where the BERT-based text representation  $n_i^V$  was used. For a stock  $j$ , the vector was obtained as a weighted average of  $n_i^V$  for which the text mentioned the stock or company. Here, the weight of article  $i$  was defined as follows:
$$
\eta_ {i} \equiv \frac {(\# \text {o f m e n t i o n s o f} j \text {i n} i)}{(\# \text {o f m e n t i o n s o f a l l s t o c k s i n} i)}.
$$
Stock embeddings:  $\Sigma$  was the cosine matrix of the stock embeddings.
Figure 3: The process of portfolio generation and evaluation over several years. The vertical axis indicates the times when the portfolio is renewed, and the horizontal axis indicates the data grouped yearly. An average of the realized annual gains is computed to evaluate the portfolio's performance.
The portfolio evaluation was conducted in a yearly setting, as illustrated in Figure 3. At the beginning of each year, given some expected gain  $E$ , the portfolio was computed by using all news articles and historic prices until the end of the previous year. In other words, for each year, the training set in the experiment consisted of samples strictly earlier than those constituting the test set. Therefore, the evaluation was conducted in a prediction setting. Then, investments were made according to the yearly renewed portfolio as in Figure 3; that is, capital was allocated to stocks according to  $w$ . The realized annual gain of the portfolio followed this
$$
\text {a n n u a l} = \sum_ {j = 1} ^ {J} w _ {j} \left(\frac {p _ {\text {e n d - o f - y e a r}} ^ {j}}{p _ {\text {b e g i n - o f - y e a r}} ^ {j}} - 1\right),
$$
where  $w_{j}$  is the proportion of investment in stock  $j$ , and  $p^j$  is the price of  $j$ .
In this way, for each of the WSJ and R&B, we obtained results over 16 and 7 years, respectively. For different expected gains  $E \in \{0.05, 0.06, \dots, 0.29\}$ , which cover typical cases in real-world portfolio construction, the average annual gain was computed.
Figure 4 shows the experimental results. The upper graphs show the annual gain with respect to different values of  $E$  (horizontal axes) for (a) the WSJ and (b) the R&B, averaged over years. Every curve corresponds to a different definition of  $\Sigma$ . It can be seen that the proposed stock embedding method outperformed the other methods, except for larger  $E$  with  $\mathrm{WSJ}^8$ . Especially for the R&B dataset, stock embedding greatly outperformed all other methods at all  $E$ .
The lower bar graph summarizes the overall aggregate gain for each method. The values in the bars indicate the average realized annual gains, while those above the bars are the ratios of the gains in comparison with that of the standard covariance method (in blue). The leftmost two bars in each bar graph show the gains of the S&P 500 portfolio and the S&P  $89^{*} / 50^{*}$  portfolio, respectively. As described above, the S&P 500 portfolio consisted of an average of around 500 stocks traded in the US, while the S&P  $89^{*} / 50^{*}$  portfolio, which was calculated with the same method but on a smaller set of stocks (89 for the WSJ, and 50 for the R&B), achieved higher gains than its S&P 500 sibling did. The values of the S&P portfolios generally went up during the periods of both datasets, and therefore, the gains were positive.
The dashed horizontal line in each bar graph indicates the result for the standard covariance method as a baseline. Its gains were only  $12.5\%$  and  $12.7\%$  for the WSJ and R&B, respectively, but with stock embeddings, the gains increased to  $17.2\%$  and  $35.5\%$ , or 1.37 and 2.80 times greater than the baseline results, respectively. This per
Our method did not perform well only for large  $E$ . The mean-variance minimization model has been reported to become unstable under the two conditions of large  $E$  and low overall market gain (Dai and Wang, 2019). The return of the WSJ period (2000-2015) was lower than that of the R&B period (2006-2013), and therefore, these two conditions were more likely to be met for WSJ.
(a)
Figure 4: Expected and realized annual portfolio gain in the investment simulations on both datasets: (a) results on the WSJ dataset (2000-2015), and (b) results on the R&B dataset (2006-2013).
formance largely beat all other variants and gives evidence of how well the stock embeddings integrated both price and textual information.
The results for the method that integrated the covariance matrix and stock embedding (in green) did not much outperform the baselines. A possible reason is that the stock embedding had already integrated the price information. As for the other variants based on pure text (in purple, orange, and brown), the results improved slightly. Among them, weighted BERT outperformed the other methods for both datasets. This indicates the potential of BERT and other recent neural language models for portfolio optimization.
This paper has proposed the idea of a stock embedding, a vector representation of a stock in a financial market. A method was formulated to acquire such vectors from stock price history and news articles by using a neural network framework. In the framework, the stock embedding detects news articles that are related to the stock, which is the essence of the proposed method. We trained stock embeddings for the task of binary classification of stock price movements on two different datasets, the WSJ and R&B. The improvements in classification accuracy with our framework, due to the clas
sifier sharing and dual-vector text representation proposed in this paper, implied that the stock embeddings successfully incorporated market knowledge from both the news articles and price history.
Because the stock embedding is a vector that can be separated from the other components of the classification model, it can be applied to other tasks besides price movement classification. As an example, we showed the use of stock embeddings in a portfolio optimization task by replacing the risk matrix in the portfolio objective function with a cosine matrix of stock embeddings. In investment simulations on the R&B dataset, our stock embedding method generated 2.80 times the annual return obtained using the covariance matrix of the historic return series. This significant gain suggests further potential of our stock embedding for modeling the correlation among stocks in a financial market, and for further applications, such as risk control and asset pricing.
We sincerely thank the anonymous reviewers for their comments. This paper was supported by the Research Institute of Science and Technology for Society (HITE 17942497), and by the University of Tokyo Gap Fund Program. The paper reflects the view of the authors only.
3361
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Sanjoy Basu. 1977. Investment performance of common stocks in relation to their price-earnings ratios: A test of the efficient market hypothesis. The journal of Finance, 32(3):663-682.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Johan Bollen, Huina Mao, and Xiao-Jun Zeng. 2011. Twitter mood predicts the stock market. J. Comput. Science, 2(1):1-8.
Deli Chen, Yanyan Zou, Keiko Harimoto, Ruihan Bao, Xuancheng Ren, and Xu Sun. 2019. Incorporating fine-grained events in stock movement prediction. In Proceedings of the Second Workshop on Economics and Natural Language Processing, pages 31-40, Hong Kong. Association for Computational Linguistics.
Zhifeng Dai and Fei Wang. 2019. Sparse and robust mean-variance portfolio optimization problems. Physica A: Statistical Mechanics and its Applications, 523:1371 - 1378.
Michal Daniluk, Tim Roctäschel, Johannes Welbl, and Sebastian Riedel. 2017. Frustratingly short attention spans in neural language modeling. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Xiao Ding, Yue Zhang, Ting Liu, and Junwen Duan. 2015. Deep learning for event-driven stock prediction. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, pages 2327-2333.
Xiao Ding, Yue Zhang, Ting Liu, and Junwen Duan. 2016. Knowledge-driven event embedding for stock prediction. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2133-2142, Osaka, Japan. The COLING 2016 Organizing Committee.
Eugene F. Fama. 1970. Efficient capital markets: A review of theory and empirical work. The Journal of Finance, 25(2):383-417.
Gabriel Pui Cheong Fung, Jeffrey Xu Yu, and Wai Lam. 2002. News sensitive stock trend prediction. In Advances in Knowledge Discovery and Data Mining, 6th Pacific-Asia Conference, PAKDD 2002, Taipei, Taiwan, May 6-8, 2002, Proceedings, pages 481-493.
Ziniu Hu, Weiqing Liu, Jiang Bian, Xuanzhe Liu, and Tie-Yan Liu. 2018. Listening to chaotic whispers: A deep learning framework for news-oriented stock trend prediction. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018, Marina Del Rey, CA, USA, February 5-9, 2018, pages 261-269.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
Ilya Loshchilov and Frank Hutter. 2017. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
Christopher D. Manning and Hinrich Schütze. 2001. Foundations of statistical natural language processing. MIT Press.
Harry Markowitz. 1952. Portfolio selection. The journal of finance, 7(1):77-91.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 3111-3119.
Alexander Miller, Adam Fisch, Jesse Dodge, Amir Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1400-1409, Austin, Texas. Association for Computational Linguistics.
Thien Hai Nguyen, Kiyoaki Shirai, and Julien Velcin. 2015. Sentiment analysis on social media for stock movement prediction. Expert Syst. Appl., 42(24):9603-9611.
Jane A Ou and Stephen H Penman. 1989. Financial statement analysis and the prediction of stock returns. Journal of accounting and economics, 11(4):295-329.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of bert. In Advances in Neural Information Processing Systems 32, pages 8592-8600. Curran Associates, Inc.
William F Sharpe. 1964. Capital asset prices: A theory of market equilibrium under conditions of risk. The journal of finance, 19(3):425-442.
Qiang Song, Anqi Liu, and Steve Y. Yang. 2017. Stock portfolio selection using learning-to-rank algorithms with news sentiment. Neurocomputing, 264:20 - 28. Machine learning in finance.
Allan Timmermann and Clive WJ Granger. 2004. Efficient market hypothesis and forecasting. International Journal of forecasting, 20(1):15-27.
Yumo Xu and Shay B. Cohen. 2018. Stock movement prediction from tweets and historical prices. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1970-1979, Melbourne, Australia. Association for Computational Linguistics.
Linyi Yang, Zheng Zhang, Su Xiong, Lirui Wei, James Ng, Lina Xu, and Ruihai Dong. 2018. Explainable text-driven neural network for stock prediction. In 5th IEEE International Conference on Cloud Computing and Intelligence Systems, CCIS 2018, Nanjing, China, November 23-25, 2018, pages 441-445.
Jiajun Zhang, Shujie Liu, Mu Li, Ming Zhou, and Chengqing Zong. 2014. Bilingually-constrained phrase embeddings for machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 111-121, Baltimore, Maryland. Association for Computational Linguistics.