Master’s Thesis 要旨

Vector Representation of Stocks from the Perspective of Natural
Language and Price History（自然言語と価格に基づく株のベクトル表現）
杜鑫

Dept. of Mathematical Informatics 48186218

田中 久美子 教授

Supervisor

1 Introduction
News articles inﬂuence the dynamics of ﬁnancial markets. Past works developed various deep learning
architectures for news-driven stock price prediction,
but have seen few gains. Meanwhile, they did not
oﬀer a way to generalize the mutual eﬀects learned
from the text and price data for other uses.
This thesis takes a new approach by explicitly describing this mutual eﬀects in terms of a vector, calledstock embedding. A stock is represented by a vector
so that its inner product with a news text vector produces a larger value when the text is more related tothe stock.
Two major advantages come from such a vector
representation of stocks. First, it concentrates thestock-speciﬁc knowledges and enables the sharing ofother parameters across stocks. This is an important
advantage to alleviate data sparseness and prevent
overﬁtting. Second, stock embeddings are portable.
They are easy to put to use in various ﬁnancial applications other than prediction. We show an example of portfolio optimization, which is the ﬁrst application of natural language processing (NLP) to themodern portfolio theory to the best of our knowledge. In this example, our method achieved 2.8x
times more gains compared to the original method
using only price data.
A part of this work has been sent to ACL 2020 [1].

2 Past Works
The idea of stock embeddings in this thesis comesfrom NLP, where vectors are trained to representwords [6], sentences [5] and even full articles [2]. The
geometry of such an embedding system usually contains rich semantic information. For example, twoword embeddings with large cosine values are often
semantically close to each other. The acquisition ofstock embeddings in this thesis is based on this original idea.
The framework for acquiring the stock embeddingsin this thesis is directly based on the work in [3, 4].
Compared to these works dedicated to price movement prediction, my thesis concentrates on the generalization and representation of the learned knowledge, and its application to other ﬁnancial usage.
As one such application, the stock embeddings areevaluated in terms of portfolio optimization. The

mean-variance minimization portfolio model proposed by Markowitz [7] was used in my work. To thebest of my knowledge, this is the ﬁrst paper applying
NLP techniques to the modern portfolio theory.

3 Text-driven price movement classiﬁcation
The text-driven price movement classiﬁcation task isused to acquire such stock embeddings. The task isdeﬁned as follows:
1 ∑∑ j(y log ŷtj +(1−ytj ) log(1−ytj )),JT j=1 t=1 t
Jmin loss = −fT

where{ŷtj ≡ f (N[t−d1 ,t+d2 ] ), ytj ≡1 if pjt > pjt−10 otherwise.

f is typically a neural network that processes news
texts and outputs the probability of ytj = 1. Whend2 < −1, this is called a prediction because no news
articles released later than t were used. This task
has been shown to be very HARD according to theeﬃcient market hypothesis. Experiment results also
showed that a neural network can barely learn anything from price and text on this prediction task.
In this thesis, I set d1 = 4, d2 = 0, so that news articles after a price movement were also incorporated.
That is, ŷtj ≡ f (N[t−4,t] ). Note this task is NOT aprediction, but referred to by text-driven price movement classiﬁcation. Even though this setting can not
be directly used for price prediction, it is still meaningful because the knowledge learned from the data
can be extracted and put to other uses.

4 Acquisition of Stock Embeddings
The learning of stock embeddings is based on the following idea. Intuitively, we want the inner productscore between a stock embedding and the vector representation of a news text, i.e. scorei,j = ⃗sj · ⃗ni , toproduce a high value when the news is related to thevector. A market status vector with respect to stock
j on day t, i.e. mjt , is then deﬁned as a weighted
average of the news vectors on day t:∑exp(scorei,j )⃗nijm⃗ t = ∑⃗ni ∈Nt⃗ni ∈Nt exp(scorei,j )where the news vectors ⃗ni were acquired via BERT[2], a SOTA text encoder. The market status vectors

Vector Representation of Stocks from the Perspective of Natural Language and Price History（自然言語と価格に基づく株のベクトル表現） (杜 鑫)within [t − 4, t] were inputted sequentially, noted byj
M[t−4,t], into a recurrent neural network with gatedrecurrent units (GRU) followed by a multi-layer perceptron (MLP):
jŷtj = MLP(GRU(M[t−4,t])).

In the experiment, the stock embeddings were learnedon two news article datasets, the WSJ and R&B.
By introducing stock embeddings, the classiﬁcation
accuracy of price movements were signiﬁcantly enhanced, as in Table 1, which suggests that the stock
embeddings had learned stock-speciﬁc knowledges.
Table 1. Mean classiﬁcation accuracy (with standard deviations in parentheses) over 10
replications, in percentage (%).
Method for mjt
previous work in [3]previous work in [4]proposedWSJ
52.1(2.51)
54.3(0.99)
60.1(0.91)

By replacing the Σ by Σcos , we can get anotherset of investment proportions, which forms a diﬀerent portfolio. Investment simulations were conductedon the two datasets, in which portfolios are generated
yearly, and the averaged annual gains are evaluated.
Figure 1 shows the averaged real annual return of theportfolios from diﬀerent methods in the R&B dataset.
Among them, the S&P 500 Index is a popular portfolio containing 505 stocks. The covariance method
corresponds to Σ, the Weighted BERT corresponds toa pure text-driven method based on BERT [2], andthe stock embedding corresponds to the Σcos which isproposed in this thesis. It is observed that the stock
embedding-based method achieved a 2.8x more gains
compared to the covariance method.

R&B
52.2(0.41)
57.2(1.08)
68.8(1.67)

5 Application of Stock Embeddings toPortfolio Optimization
A portfolio is essentially a set of investment proportions w = [w1 , . . . , wJ ]T assigned to J stocks. A
good portfolio usually has a high gain and low uncertainty (risk). Given the prices of stock j at timet and t − 1, the return of the stock can be computed by rjt = ptj /pt−1j . The portfolio’s return is then∑Jrt = j=1 wj rjt , with the variance
Var(rt ) =J ∑J∑wj wj ′ cov(rjt , rjt ′ ).

j=1 j ′ =1Let w = [w1 , . . . , wJ ]T and Σ = (cov(rˆ jt , rjt ′ ))J×J
where cov
ˆ is the estimated covariance, then we canconstruct a portfolio w with expected portfolio return E and minimized variance (risk) by solving theproblem below:
minw

subject toVar(rt ) = wT Σw(2b)T(2c)w 1 = 1,0 ≤ wj ≤ 1

j = 1, ..., J,(2d)()Twhere 1 = 1, 1, . . . , 1
is a J-dimension column
vector consisting of 1’s. Note that higher E usuallymeans higher risk born by the investor.
Since stock embeddings reﬂects the characteristics
of stocks in response to news articles or the ﬁnancial events behind, their cosine values can be used
to measure the correlation among stocks. For eachpair of stocks, the cosine of their stock embeddings
is calculated, which forms a cosine matrix Σcos :Σcosj,j ′ := cos(sj , sj ′ )6 Conclusion
This thesis proposed a method to extract stock embeddings from both price history and news article
texts. This is done by use of the task text-driven
price movement classiﬁcation. The higher accuracies
achieved by stock embeddings demonstrated that thestock embeddings have learned stock-speciﬁc knowledges from the text and the price. Moreover, a further experiment of applying the stock embeddings
to portfolio optimization strongly suggests the potentials of the stock embeddings in ﬁnancial applications, including portfolio optimization.

Bibliography(2a)Tw r̂ = E,Fig. 1. Averaged real annual gains from diﬀerent
methods on R&B (2006-2013).

[1] Xin Du and Kumiko Tanaka-Ishii. Stock embeddings
acquired from news articles and price history, and anapplication to portfolio optimization. Submitted toACL 2020.
[2] Devlin et al. BERT: Pre-training of deep bidirectionaltransformers for language understanding. In NAACL,pages 4171–4186, June 2019.
[3] Ding et al. Deep learning for event-driven stock prediction. In IJCAI, pages 2327–2333, 2015.
[4] Hu et al. Listening to chaotic whispers: A deep learning framework for news-oriented stock trend prediction. In ACL, pages 261–269, 2018.
[5] Lin et al. A structured self-attentive sentence embedding. In ICLR, 2017.
[6] Mikolov et al. Distributed representations of words
and phrases and their compositionality. In NeurIPS,
pages 3111–3119, 2013.
[7] Harry Markowitz. Portfolio selection. The journal ofﬁnance, 7(1):77–91, 1952.