A Collaborative Attention Adaptive Network for Financial Market Forecasting
Authors: Qiuyue Zhang
Yunfeng Zhang
Fangxun Bao
Caiming Zhang
Peide Liu
Xunxiang Yao
Venue: ICLR 2022 Submitted (2022)
Keywords: Financial market forecasting
Deep fusion
Collaborative attention

Abstract: Forecasting the financial market with social media data and real market prices is a valuable issue for market participants, which helps traders make more appropriate trading decisions. However, taking into account the differences of different data types, how to use a fusion method adapted to financial data to fuse real market prices and tweets from social media, so that the prediction model can fully integrate different types of data remains a challenging problem. To address these problems, we propose a collaborative attention adaptive Transformer approach to financial market forecasting (CAFF), including parallel extraction of tweets and price features, parameter-level fusion and a joint feature processing module, that can successfully deeply fuse tweets and real prices in view of the fusion method. Extensive experimentation is performed on tweets and historical price of stock market, our method can achieve a better accuracy compared with the state-of-the-art methods on two evaluation metrics. Moreover, tweets play a relatively more critical role in the CAFF framework. Additional stock trading simulations show that an actual trading strategy based on our proposed model can increase profits; thus, the model has practical application value.

Under review as a conference paper at ICLR 2022
A COLLABORATIVE ATTENTION ADAPTIVE NETWORK FOR FINANCIAL MARKET FORECASTING
Anonymous authors Paper under double-blind review
ABSTRACT
Forecasting the financial market with social media data and real market prices a valuable issue for market participants, which helps traders make more appropriate trading decisions. However, taking into account the differences of different data types, how to use a fusion method adapted to financial data to fuse real market prices and tweets from social media, so that the prediction model can fully in- tegrate different types of data remains a challenging problem. To address these problems, we propose a collaborative attention adaptive Transformer approach to financial market forecasting (CAFF), including parallel extraction of tweets and price features, parameter-level fusion and joint feature processing module, that successfully deeply fuse tweets and real prices in view of the fusion method. Extensive experimentation is performed on tweets and historical price of stock market, our method can achieve better accuracy compared with the state-of-the- art methods on two evaluation metrics. Moreover, tweets play relatively more critical role in the CAFF framework. Additional stock trading simulations show that an actual trading strategy based on our proposed model can increase profits; thus, the model has practical application value.
INTRODUCTION
In recent years, the use of social media information to predict the financial market has attracted the attention of more and more researchers, and some satisfactory experimental results have been achieved. This is due the fact that social media information contains investor-related attitudes and subjective sentiments to the financial market, resulting in many investment banks and hedge funds trying to dig out valuable information from media information to help better predict financial markets, which play a key role in predicting the market. At the same time, the efficient market hypothesis (EMH) introduced by /ama 1970)) points out that the current price of the asset reflects all the prior information that is immediately available. Therefore, using social media information and the actual price of the current financial market seems to be able to complete the market forecasting more accurately.
To the best of our knowledge, the different types of data fusion methods commonly used in the field of financial market forecasting mainly concatenation or weighted sums. They are simple, direct, and almost no parameter exchange methods to achieve data fusion (Xu & Cohen 2018 Perez- Rua eet al. 2019 Xu et al. 2019). In the field of computer vision (CV), multimodal fusion methods mainly used for the fusion of data based on text and images. By realizing the interaction of differ- information, extracting rich feature information, and obtaining satisfactory experimental results (Baltrusaitis et al., 2019). It can be seen that the multimodal fusion method in the CV field seems be more conducive to feature exchange and feature supplementation than the current method in the financial field (Lu et al. 2018 Kim et al. 2017). Therefore, we can infer that choosing an adap- tive financial data fusion algorithm to achieve the integration of social media information and actual market prices is effective for improving the forecasting effect of the financial market. However, the current financial market forecasting field basically does not consider too many modal fusion meth- ods, which may be one of the reasons that limit the current forecasting effect. In addition, the role of text and historical prices in the fusion method is different from multimodal fusion tasks, such as face recognition, medical aided diagnosis, and visual analysis. The fundamental reason is that the application scenarios are inconsistent, the source data for other tasks objective, and the financial data is affected by the sentiment of market investors. Taking into account the difference between

---

Under review as a conference paper at ICLR 2022
tweets and prices, we should pay attention the effects of both when choosing a fusion method, and use the quantitative analysis of experimental results.
To effectively solve the above problems, we propose collaborative attention (co-attention) Trans- former approach adaptive to financial market forecasting called CAFF, partially inspired by the re- cent proposed multimodal fusion in bidirectional encoder representation from Transformers (BERT), which was originally developed to utilize Transformer to realize the feature interaction between text and image data (Lu al. 2019). To verify the rationality of the CAFF method, included the existing traditional multimodal fusion methods in the experiments for comparison.
In summary, the contributions of our work are as follows:
To the best of our knowledge, this work is among the first to introduce a co-attention Trans- former fusion approach adaptive to financial market forecasting, which is inspired by the multimodal fusion method and takes into account the characteristics of financial data.
We propose a novel financial market forecasting framework based on the idea deep fu- sion model multisource data analysis. The components of the framework work together to extract tweets and price features in parallel, fuse the features and realize accurate predic- tion.
Experimental results on stock market forecasting tasks demonstrate that our proposed method achieves substantial gain over state-of-the-art methods. Moreover, the results also reveal that under the CAFF fusion framework, the quantitative analysis results show that social media information has played a relatively more critical role.
2 RELATED WORK
Some researchers have adopted the method of fusing real market prices and textual information reflecting investor sentiment from social media. In this section, we review the work related to financial market forecasting that fuse prices and text.
A variety of analysis approaches based on text and prices to market forecasting have been proposed in the research literature. For example. Xu & Cohen (0088) proposed new depth generation model for the stock market forecasting task, which combines text and price signals as the source infor- mation. Similarly, Li et al. (2020) also used the concatenation method to integrate the sentiment information contained in the news data and stock prices to predict the Hong Kong stock market. In addition, Xu al. (020)) proposed stock movement prediction network based on tweet and stock prices by means of incorporative attention mechanism that combines local and context attention mechanisms through incorporative attention to clean up context embedding using local semantics. The approach also makes use of concatenation. It can be seen that the common fusion methods in the field of financial market prediction are still surface feature fusion without parameter interaction. At present, the mainstream fusion technology in the field of CV has attracted our attention, such as bilinear pooling, attention- based fusion, in which bilinear pooling creates a joint representation s- pace by calculating the outer product to facilitate the multiplicative interaction between all elements in the two vectors, which is obviously not suitable for describing text and representing the value of price changes. Lu et al.] (2019) proposed the visual language BERT (ViLBERT) model, which extends the Transformer to the fusion operation of processing images and texts. These extracted features can interact through a parallel attention layer, which provides creative way for feature op- timization, but we have to consider the differences between financial data and CV, speech task and other fields. Therefore, developing a method suitable for financial data with the help of the existing fusion methods is a problem worthy of discussion.
In summary, using an improved deep learning framework based on a fusion algorithm to analyze fi- nancial data to achieve financial market forecasting appears to be effective, which can be confirmed via quantitative analysis. On the basis of the above advanced theoretical analysis, our research mo- tivations are follows. First, we believe that framework that processes text and prices in parallel and fully fuses them would perform better than a single data source, where the fusion algorithm plays a key role. However, referring to the existing fusion methods in the CV field, the most natural and effective way to process prices may be to consider the original form as 1-D data rather than as a 2-D matrix image Gupta et al. (2020). Moreover, the role of text used to describe the movement of
2

---

Under review as a conference paper at ICLR 2022
financial market is inconsistent with prices, which are different from the text and image in the modal task of CV fields. The above problems require an applicable prediction framework based on the fusion method to realize the feature extraction and fusion of tweets and prices with temporal- ity. as well as a matching activation function. Hence, we propose a new framework called CAFF, which uses the Transformer model adapted financial data and variety of attention mechanisms to collaboratively predict the movement on the target trading day.
MODEL OVERVIEW
We propose CAFF, which conforms the characteristics of financial data, and the complete frame- work is shown in Figure [ The upper of the model deeply extracts text features to learn the text representation of tweets, and the lowers extracts market price features in parallel. The operation with Transformer as the main structure in the middle is used to fuse tweets and price information, and the right side is used to process the merged features and realize the prediction of market trends on the target trading day.
3.1 THE PROPOSED MODEL: CAFF
Considering that there is certain lag in the target trading day d, allowed simulate and predict other trading days close to in principle. We can not only predict the trend of the target trading day itself, but also the trend of other trading days during the lag period. For example, if choose 08/06/2021 as the target trading day, then 03/07/2021 and 07/06/2021 represent the endpoints of the lag period (the lag period is usually days); thus, we capture the relationships between the predictions mainly within this sample range. However, considering the actual conditions of the financial market, we neglected nontrading days in the calculation process to achieve the effect of effectively organizing and utilizing input data. In general, ,
predict series of movements [11 ,.,, 2T], where the target trading day is 2т and the rest are auxiliaries.
2
3.1.1 INPUT REPRESENTATION
Next, we introduce the processing of text and stock prices to obtain the input representation. Text data is regarded feature extraction task NLP domain. Considering the inherent correlation and volatility, we adopt relatively direct method to deal with stock prices to maintain the original structure.
Feiterr Fadis
Traper al atterian
ABL Nem
End Prewar
PLESS Coon
Torai
Priee Satorna
Figure 1: The overall architecture of the proposed stock movement prediction model CAFE
Tweet-level Model: Generally, tweets posted on social platforms for given stock often contain multiple inconsistent items on a given trading day. To learn more profitable and valuable informa- tion from large number of investor tweets, adopt the Bi-LSTM and self-attention to obtain the representation as the input of the fusion component. For each sentence t,, we adopt the pre- trained word embedding (GloVe) to project each word tag the d dimensional space and the
3

---

Under review as a conference paper at ICLR 2022
pretrained word embedding as the input of Bi-LSTM.
ft = o 1h-11, [.]+ + by)
(1)
4 oWW, [1 b.)
č tanh(Wa (1 + bc)
C. = ft = C--1 + it Ct
O. oWo [2 bo)
ht Og tanh(C.))
h h [he, hdl
In accordance with the nature of time series data, we use an RNN with Bi-LSTM units and a self- attention mechanism recursively extract features. The design characteristics of LSTM allow to model text data (e.g., tweet data, news data) to extract text features and capture the context rela- tionship. Bi-LSTM adds process of forward and backward concatenation of hidden vectors on the basis of the realization of LSTM. Formulas (17 describe the details of Bi-LSTM.
The essence of the attention function can be described as query (q) to a series of key-value (k-v) mappings. In current research, the k and are often the same, that is, key = value. Formulas (8) 10 describe the details of the attention operations. The only difference between the self-attention mechanism and the abovementioned conventional attention mechanism is that g k. Self-attention mechanisms have become a hot topic in recent research and have been explored in different tasks. The main reason choose the self-attention mechanism after Bi-LSTM is that the self- aiff-attention is calculated for each word and all words, so regardless of the distance between them, long-distance dependency relationships be captured to achieve the contextual interaction of tweets to obtain key text features.
ar; = so/tmaz((Fkeep) 4))
(8)
att((KV)).) :.N ax
(9)
Allention((KKV), =dll((K (), ... att((K, V), (M)
(10)
Price-level Model: Stock prices more intuitively reflect the real market conditions, while tweets contain objective investor attitudes, which allows the to complement each other. However, the movement of stock has random volatility and is determined by continuous changes in prices rather than the absolute values of the opening and closing prices. Thus, the original stock price vector of trading day d is not fed directly into the neural network; we employ normalization strategy to obtain an adjusted closing price. The price adjustment formula is shown below:
Pa = På p/a P'al
(11)
Pd
Pa
(12)
P4-1
where Fha 1/2 and bl2 denote the closing price, highest price and lowest price vectors, respectively.
4

---

Under review as a conference paper at ICLR 2022
3.1.2 FUSION METHOD BASED ON CO-ATTENTION TRANSFORMER
Due the excellent performance, an increasing number of Transformer-basee frameworks have been proposed to improve various tasks. The input is divided into three parts, which is similar to the self-attention mechanism we mentioned above. In fact, Transformer is composed mainly of the attention mechanisms, where the multihead attention mechanism is the parallel of multiple atten- tion mechanisms, allowing the model to pay attention to information from different representation subspaces at different locations. The follow-up is to input the result of the Add & Norm into the feedforward neural network (FNN) in the fully connected layer.
We improve the fusion method upon this framework to accommodate financial data. Next, we elab- orate the complete fusion process in detail. To achieve deep interaction, we first create the q, k and vector; that is, the input representations generated by text embedding and stock price are repre- sented and the Query (Q), Key (K) and Value (V) matrices are generated by multiplying the input representations and the three generated matrices in the training process. Moreover, the conversion process adopts the GELU activation function, which introduces the idea of random regularization. This approach represent a more intuitive and natural understanding of the probability description of the neuron input, which matches a random value of 0 or I for the input. Furthermore, some exper- iments have shown that GELU is better than ReLU and ELU in CV, NLP, and speech tasks. The approximate expression of the GELU is as follows:
Gelu(x) 0.5 + tanh (/ah//2//T(//// 0.044715x
(13)
Scaled dot-product attention is the main component of the multihead attention mechanism, so its calculation process is equivalent to reducing the entire implementation. First, we need to calculate the dot product between Q and K to measure the similarity and then divide the value by /dz to prevent the result from being excessively large, where d, and dg are the dimensions of the vectors. Then, the result is normalized by matrix V to obtain the attention vector. The matrix of outputs is as follows:
probability distribution via the softmax function and multiplied
QKT
Attention K, V) = softmar W
(14)
The above calculation process mentions only single-layer attention, while multihead attention is used for parallel calculations. Then, all the values are concatenated to reduce the loss by decreasing the dimensionality. Therefore, we linearly project the queries in different ways with the keys and values h times to learn the linear projections of dy. de and d,, Next, apply the attention function in parallel on each of these queries, keys and values to generate d, dimensionnl output values. The above single values are concatenated in series and projected form the final required value. By means of the above process, multihead attention allows the model to jointly focus on information from different locations to achieve deep interaction of internal information. The multihead attention function is as follows:
MultiHead(( Q, K, V)- Concat(head, headz, ... heads) W
(15)
where OO represents the weight matrix. In this work, we employ h 5 parallel attention layers or heads.
Add & Norm denote residual connection and layer normalization, where residual connection has a better effect on deeper neural networks. When the network layer is very deep, the spread of values decreases with the weight. Layer normalization prevents abnormal values due excessively large or small positions in a certain layer and is applied in training problems when returning the gradient of the neural network to ensure the stability of training. Add & Norm are applied after each subnetwork to make the deep neural network training smoother. Each submodel in the encoder contains a fully connected FNN layer that acts equally on each position. The layer is composed of two linear transformations separated by GELU activation.
FFN = max xW + b²
(16)
Although the linear transformation operations used at all positions are the same, exclusive optimal parameters are used between layers.
5

---

Under review as a conference paper at ICLR 2022
3.1.3 JOINT FEATURE PROCESSING: LSTM
The first step of LSTM is to determine that unnecessary information should be discarded from the cell state, which is realized by the sigmoid layer called the "forgetting door". This layer considers h4-1 (previous output) and 21 (current input) and outputs a value between and for each number in the cell state Ct-1 (previous state). The next step is to determine the information to store in the unit state and to output the information to complete the operation. Formulas (66 describe the details of these LSTM operations.
LSTM substantially improves the use of RNN, and the emergence of attention models has led to great progress. Therefore, we further analyze the joint information through temporal attention to achieve the prediction task.
3.1.4 JOINT FEATURE PROCESSING: TEMPORAL. ATTENTION
We introduce improved temporal attention mechanism to realize stock prediction the target trading day and auxiliary trading days. Although different types of temporal attention mechanisms have been proposed, we need method that is compatible with text and stock price data, that is, method that accounts for the relationship and key information of both types of data. To this end, as shown in the complete CAFF framework, the temporal attention mechanism is composed of two main parts, namely, the dependency score and information score, which are, respectively, used to obtain the dependency of stock data and more favorable key information.
dr- tanh (Walz, h4] + ba)
(17)
where Wa and bu denote the weight matrix and bias, respectively. Then, the dependency score and information score obtained by nonlinear projection of d,, and the calculation method is shown in formulas (0).
o wi tanh(Wa,
(18)
2/ = Gtanh
(19)
softmaz(1) @ ('m)
(20)
where Wa, and Wa.m are weight parameters and is the set of all auxiliary trading days. Our
goal is to predict the movement on the target trading day, dr is used the final information. To achieve the final prediction, the softmax function is used to obtain the Up or Down prediction for the target trading day. 2 = softmar(w,,de + by), <
(21)
2T softmar( wr [Zrr, drl by)
(22)
where wy and wr weight parameters and by and bt are biases. The above calculation steps can achieve a complete prediction of stock market movement on the target trading day.
EXPERIMENTS AND DISCUSSIONS
4.1 DATASET
To verify the actual performance of CAFF and related methods of stock prediction, we adopt stock price and tweets datasets. Stock traders often post personal opinions on social platforms such as Twitter to express their inner practical ideas. On the basis of the heat of discussion, the 88 stocks with the highest capital scale rankings were selected for comparison experiments.
Some stocks show only minimal rates of change, so we implement relevant measures to avoid affect- ing the experimental results. We chose the upper and lower limits of stock price changes proposed by Hu et al.] (2018) for stock trend prediction. Since the stock movement prediction task can be regarded as a binary classification problem, we set two thresholds: -0.5% and 0.55%. For samples with movement ratios of < -0.55% and >0.5%% 0 and were used represent the fall and rise, respectively, and 38%72% of the target is removed between the two thresholds. According to the two thresholds set above to balance the two predicted categories, 26614 prediction targets are obtained for the entire dataset, of which the two categories account for 49.784 78% and 50. 50.22%, respectively. We divide the dataset into three parts, namely, the training set, test set and validation set, by date.
6

---

Under review as a conference paper at ICLR 2022
Table I: Performance of baseline methods and CAFF in ACC and MCC. Model
Accuracy (%)
MCC
ARIMA Adebiyi et al. 2014
51.39
-0.0205
TSLDA Nguuyen & Shiral
2015
54.07
0.0653
HAN (Hu et al. 2018
57.14
0.0723
CH-RNN Wu al. 201 8
59.15
0.0945
StockNet (Xu & Cohen
2018
58.23
0.0807
Adv-LSTM (Feng et
2019
57.20
0.1483
CapTE (Liu et al.
201
64.22
0.3481
SMPN [uu al.
2020
59.74
0.1586
CAFE. CON
60.19
0.2436
CAFF SUM
59.64
0.1589
CAFF, BIL
58.49
0.1377
CAFF, TRA
57.47
0.1163
CAFF (W/O text)
55,48
0.1657
CAFF (W/O price)
64.89
0.3288
CAFF faggsizee_7
63.77
0.3109
CAFF (lag size si0)
62.50
0.2258
CAFF
65.78
0.3792
4.2 TRAINING SETUP
Regarding the specific parameter settings of the experiment, the most basic operation is to obtain the initial word embedding of the text through the GloVe pretraining model, and the word embedding size is 50. The maximum numbers of messages and words in single message are set to 30 and 40, respectively. The traditional Adam optimizer is more line with framework this paper and is used to train the model with an initial learning rate of 0.001. In addition, the decay rate is set to 0.96, and the decay step set to 100. We adopt conventional batch operations during the training process, and the batch size is 32. The whole method is implemented using TensorFlow (version 1.14.0).
4.3 EVALUATION METRICS
The model predicts the upward or downward movement of stocks for the next day. In this work, we follow some previous research work to evaluate stock prediction (Xie et al. 2013), and standard metrics, including accuracy (ACC) and the matthews correlation coefficient (MCC), performance evaluation indicators. The MCC is essentially the correlation coefficient between the observed and predicted binary classes. As shown in formula (23), lp. fp, and fn represent the numbers of samples classified as true positives, false positives, true negatives, and false negatives, respectively. The MCC is calculated follows:
used as
tp x tn fp X fn
MCC
(23)
V + fn) + (p) fn)
4.4 VERIPYING THE EFFECTIVENESS OF CAFF
4.4.1 COMPARISON WITH BASELINE MODELS
To test the diversity and differences of different model architectures, consider 8 baseline models to compare the performance of the prediction results. The baseline models are designed from the perspective of overall achievement to predict stock movements and composed of common neural networks, using text or stock prices as input data.
Table I shows that the CAFF model achieves the best performance, where numbers in bold font rep- resent the best performance score. An accuracy of 56% is generally considered to be a satisfactory result of binary stock movement prediction (Nguyen & Shirai 2015). In fact, according statistical

---

Under review as a conference paper at ICLR 2022
laws, the probability of binary random prediction is 50%. Although the ARIMA and TSLDA meth- ods in Table [I have accuracies less than 56,, the statistical models and the deep learning framework based on social media based on these methods are still relatively satisfactory.
In the stage of text feature extraction. both HAN and CH-RNN use a traditional attention mecha- nism, while CapTE is used in Transformer, which has achieved breakthrough results in NLP and is the main factor leading to the difference in the final experimental results. In addition, CapTE chose the capsule network commonly used in NLP cross-domain direction after processing the text features, which provides researchers with an efficient way to handle temporal text. Adv-LSTM used stock prices as the original analysis data, but the experimental results show that there is room for further improvement. The main reason for this phenomenon may be the range of features covered by the text and price, which may also be affected by the complete framework. The remaining two deep learning frameworks, StockNet and SMPN, both adopt the idea of fusing text and stock prices with the attention mechanism as the main body and adopt the traditional fusion method of concate- nation. However, from quantitative perspective, their results are similar but worse than those of CAFF. Therefore, it is effective for us to use a suitable fusion method and an adaptive deep learning framework for stock price prediction tasks.
4.4.2 COMPARISON WITH FUSION METHODS
This subsection provides four solutions that fuse text and stock prices and searches for the best so- lution to achieve state-of-the-an performance. The four fusion methods are concatenation, weighted summation, bilinear pooling, and Transformer, which changes the input order to quantitatively an- alyze the fusion method that is most suitable for text and stock data under the current framework, called CAFF. CON, CAFF. SUM, CAFF. BIL, CAFF. TRA respectively.
We can conclude that the fusion operation can be used to quantitatively determine which method more suitable for CAFF and judge the criticality of the source input data. Concatenation and weighted summation are the most classic algorithms for fusing different modes of information, and many studies have shown that they are effective to a certain extent. Especially in the field of stock market forecasting, concatenation is currently the most commonly used fusion method. According to the quantitative results, CAFF. CON and CAFF. SUM have achieved satisfactory results, even compared with the baseline model in the previous subsection. Bilinear pooling is feature fusion algorithm for CV proposed in recent years; a large number of researchers continue to improve the method obtain breakthrough results. However, bilinear pooling does not appear to be suitable for financial data with timeliness. The main reason may be that the extracted features are subjected to the outer product operation during fusion and the dimensionality is forced be reduced, which causes feature loss. Finally, the Transformer-basee fusion algorithm that reverses the input order has the worst performance, further confirming that the text and stock price have different effects on the final forecast results and that the text plays a key role.
4.4.3 ABLATION STUDY
To verify the integrity of our CAFF model and make more detailed analysis of the key compo- nents, we construct the following variants in addition to the fully equipped CAFF. CAFF (W/O text) means that the input data do not include the text corpus; that is, only the historical stock prices are considered, and CAFF (W/O price) has a similar meaning. Furthermore, we propose variables with alternative parameter values in which the length of the lag time is modified to 7 days and 10 days; these methods are called CAFF (lag_size _7) and CAFF (lag_size_1O), respectively.
Table shows the performance of CAFF and different variants. CAFF (W/O text) performs the worst, indicating that stock movement is affected by many unfixed factors. Even compared with the baseline model in Table D the performance of CAFF (W/O text) not satisfactory, which proves that historical stock prices play an auxiliary role in the prediction framework. However, CAFF (W/O price) has produced extremely competitive results, which quantitatively confirms the criticality of the text. Therefore, social media contains lot of valuable market information, and studies have shown that this type of information is useful for predicting stock prices and for other financial tasks (Diaz al. 2020).
In the ablation study, we additionally modified the lag period of the input data from the current calendar days to 7 days and 10 days. As shown in Table [ experiments with lag period of
8

---

Under review as a conference paper at ICLR 2022
Table 2: Profits comparison between CAFF and Cap'E
Stock
CAFF
Return(%)
CapTE
Return%)
Stock
CAFF
Return(%)
CapTE
Return(%)
AAPL
$1287
12.87
$1196
11.96
GOOG
$1003
10.03
$864
8.64
ABBV
8910
9.1
$729
7.29
INTC
$1847
18.47
$1200
12
BAC
$993
9.93
$941
9.41
ORCL
$985
9.85
$703
7.03
CELG
$1567
15.67
$1430
14.30
PFE
$1100
11
$1053
10.53
CVX
$1059
10.59
$1005
10.05
WMT
$1572
15.72
$1473
14.73
DIS
$892
8.92
$703
7.03
XOM
$1190
11.9
$1092
10.92
and 10 calendar days did produce better results than experiments with lag period of The reasons for this phenomenon are as follows: tweets and historical stock prices are time-sensitive, and the information utilization value is lower farther from the target trading day; moreover, investor comments are more inclined to reflect current market conditions and are not suitable for predicting long-term trading behavior.
4.5 MARKET TRADING SIMULATION
The stock simulation strategy proposed by Ding et al. (2015) to evaluate the performance is used of the CAFF method, which simulates the behavior of daily trader who uses model in simple way. CAFF is compared with the best-performing Cap'I'B to illustrate the potential value of our method. The detailed simulation strategy implemented as follows: if the model indicates that the price of single stock will rise the next day, the trader will invest $10,000 worth of that stock at the opening price. After the purchase, the length of time the trader holds the stock is determined by the actual market changes. If the stock can earn profit of 2% more in the next period of time, the trader immediately sells it: otherwise, traders sell the stock at the closing price the end of the day. In addition, if the model indicates that the price of single stock will fall, the same strategy can used to short the stock. If the trader can buy the stock price that 1% lower than the short-selling price, the trader will buy the stock cover the position; otherwise, the trader buys the corresponding stock at the closing price.
In Table 2 selected 12 representative stocks in 22 trading days (30 calendar days) to illustrate the specific profits the two models, with the maximum return exceeding 20%. The experimental results consistently illustrate that CAFF can obtain higher profits than CapTE and has a superior practical application value. In addition to the reasons analyzed in Section 4.4 we note that if there financial data the stock market the previous day, the models cannot predict the stock price trend for the day because they have no information for quantitative analysis. Not only will this scenario reduce the accuracy and MCC, will also harm actual profits. Furthermore, we take the specific market returns at that time as the benchmark, and the returns of the two models are generally higher than the average returns of the actual market. The monthly return of the historical real stock market is 8.35 %, and the average market return obtained by CAFF is 9.3 %.
CONCLUSION AND FUTURE WORK
We study a popular research topic in current financial market, that how fuse market prices and text information, and quantitatively analyze the key role of source information to enable the pre- dictive model to process different financial data in parallel in an intelligent manner. To solve these problems, we proposed collaborative attention Transformer fusion model to perform financial mar- ket forecasting using historical text and prices. The proposed method is inspired by the cross-modal fusion method and accounts for the characteristics of financial data. According to the quantitative analysis results from the above experiments, text play a key role in the overall framework and are es- sential to improving the experimental accuracy. Therefore, advanced NLP technology must be used to process text and analyze context relations. How to choose a method that suitable for analyzing time series information and has practical application value will be addressed in future research.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Studies that doesn't involve human subjects, practices data set releases, potentially harmful in- sights, methodologies and applications, pontential conflicts of interest and sponsorship, discrimi- nation/bias/faireess concerns, privacy and security issues, legal compliance, and research integrity issues.
REPRODUCIBILITY
The dataset is available on gitjchhiiggggghhhoooooyyuuuuuuse
REFERENCES
Ayodele A. Adebiyi, Aderemi O. Adewumi, and Charles K. Ayo. Stock price prediction using the ARIMA model. In Proceedings of the 2014 UKSim-AMSS 16th International Conference on Computer Modelling and Simulation, pp. 106-112. IEEE, 2014. ISBN 978 9781479949236.
Tadas Baltrusaitis, Chaitanya Ahuja, and Louis Philippe Morency. Multimodal Machine Learning: A Survey and Taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41 (2):423-4433 2019. ISSN 19393539.
Juglar Diaz, Barbara Poblete, and Felipe Bravo-Marquez. An integrated model for textual social media data with spatio-temporal dimensions. Information Processing and Management, 57(5): 102219, 2020. ISSN 03064573.
Xiao Ding, Yue Zhang, Liu Ting, and Duan Junwen. Deep Learning for Event-Driven Stock Predic- tion. In Proceedings the the Twenty-Fourth International Joint Conference on Artificial Intelligence (IJCAI 2015), pp. 2327-2333, Morgan Kaufmann, 2015.
Eugene F. Fama. Efficient capital markets: A review of theory and empirical work. The Journal of Finance, 25(2):333-41 1970.
Fuli Feng, Huimin Chen, Xiangnan He, Ji Ding, Maosong Sun, and Tat Seng Chua. Enhancing stock movement prediction with adversarial training. Proceedings of the Twenty-Eights International Joint Conference on Artificial Intelligence (IJCAI-19), pp. 5843-5849. Morgan Kaufmann, 2019.
ISBN 9780999241144 1411.
Pooja Gupta, Angshul Majumdar, Emilie Chouzenoux, and Giovanni Chierchia. SuperDeConFuse: A supervised deep convolutional transform based fusion framework for financial trading systems. Expert Systems with Applications, 169:1 14206, 2020. ISSN 09574174.
Ziniu Hu, Weiqing Liu, Jiang Bian, Xuanzhe Liu, and Tie Yan Liu. Listening to chaotic whispers: deep learning framework for news-oriented Stock trend prediction. Proceedings of the 11th ACM International Conference on Web Search and Data Mining (WSDM 2018), pp. 261-269.
ACM, 2018. ISBN 9781450355810.
Jin Hwa Kim, Kyoung Woon On, Woosang Lim, Jeonghee Kim, Jung Woo Ha, and Byoung Tak Zhang. Hadamard product for low-rank bilinear pooling. In Proceedings of the 5th International Conference on Learning Representations (ICLR 2017), pp. 1-14, 2017.
Xiaodong Li, Pangjing Wu, and Wenpeng Wang. Incorporating stock prices and news sentiments for stock market prediction: case of Hong Kong. Information Processing and Management, 57 (5):102212, 2020. ISSN 03064573.
Jintao Liu, Xikai Liu, Hongfei Lin, Bo Xu, Yuqi Ren, Yufeng Diao, and Liang Yang. Transformer- Based Capsule Network For Stock Movements Prediction. In Proceedings of the First Workshop on Financial Technology and Natural Language Processing (FiNNLP@IJCA) 2019), pp. 66-73. Morgan Kaufmann, 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining task-agnostic visiolin- guistic representations for vision-and-lan annguang tasks. Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), pp. 1-11. MIT Press, 2019.
10

---

Under review as a conference paper at ICLR 2022
Pan Lu, Hongsheng Li, Wei Zhang, Jianyong Wang, and Xiaogang Wang. Co-attending free-form regions and detections with multi-modal multiplicative feature embedding for visual question answering. In Proceedings of the 32nd AAAI Conference Artificial Intelligence, pp. 7218- 7225. AAAI, 8. ISBN 9781577358008.
Thien Hai Nguyen and Kiyoaki Shirai. Topic modeling based sentiment analysis on social medi- for stock market prediction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pp. 1354-1364. ACI., 2015. ISBN 9781941643723.
Juan Manuel Perez- Rua, Valentin Vielzeuf, Stephane Pateux, Moez Baccouche, Frederic Jurie. MFAS: Multimodal fusion architecture search. In Proceedings of the 2019 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 6959-6968, IEEE, 2019. ISBN 9781728132938.
Huizhe Wu, Wei Zhang, Weiwei Shen, and Jun Wang. Hybrid Deep Sequential Modeling for So- cial Text-Driven Stock Prediction. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM' 18), pp. 1627-1630. ACM, 2018. ISBN
9781450360142.
Boyi Xie, Rebecca J. Passonneau, Leon Wu, and Germán G. Creamer. Semantic frames to pre- dict stock price movement. In Proceedings of the 51st Anmual Meeting of the Association for Computational Linguistics, pp. 873-883. ACL, 2013. ISBN 9781937284503.
Hongfeng Xu, Lei Chai, Zhiming Luo, and Shaozi Li. Stock movement predictive network via incorporative attention mechanisms based on tweet and historical prices. Neurocompuling, 418: 326-339, 2020. ISSN 18728286.
Nan Xu, Wenji Mao, and Guandan Chen. Multi-interactive memory network for aspect based mul- timodal sentiment analysis. In Proceedings of the Thirty-Third AAAI Conference Artificial Intelligence (AAAI-19) 19), pp. 371-378. AAAI, 2019. ISBN 9781577358091.
Yumo Xu and Shay B. Cohen. Stock movement prediction from tweets and historical prices. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp- 1970-1979. ACL, 2018. ISBN 9781 07819488 1948087322.
11


Reviews:
[{"review_id": "tyJdT-Tpp7m", "reviewer": "Reviewer_TVcH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- This method is well-motivated and the problem it solves is interesting and important.\n- It achieves state-of-the-art performance in a stock prediction dataset.\n\nWeaknesses:\n- This paper utilizes most of the spaces on Page 3, 4, and 5 to describe some well-established concepts like LSTM, attention, self-attention, and Transformer layer. This leads to several potential problems.\n    - It emphasizes too much on some technical details that are not strongly related to the proposed model and contributions (e.g., the difference between GELU, ReLU, and ELU), but ignores many details of their own implementation.\n    - On the one hand, it is hard to identify the novelty of the proposed model. It would be very helpful to compare the proposed modules with existing modules such as Transformer. For example, the multi-features fusion module looks like a Transformer Encoder Layer. If it is true, it would be much better to specifically claim that \"we use a Transformer Encoder Layer for feature fusion, where the query is xxx, the key is xxx\" instead of renaming it as \"multi-features fusion\" and describe each component of it in detail. This may lead to a lot of confusion.\n    - On the other hand, it is hard to understand the technical details of the proposed model. For example, this paper describes the formula of multi-head self-attention but does not give the details of how they construct query/value/key and what the dimension of each vector is, while the first part is well-known and the second part is what this paper really propose.\n- Some minor problems/suggestions:\n    - Figure 1 and Table 2 are out of the boundary\n    - I think it would be very helpful to align the name of the sections (e.g., Sec 3.1.2) with the module names in Figure 1 (e.g., multi-features fusion). This will make it much easier for readers to align your text description with the figure.\n\nQuestions:\n- What is a training sample (e.g., an (x, y) pair) of the proposed model? Based on the description, the output (y) is a binary label that indicates the rise/down of a specific stock. What is the input (x)? Is it the combination of all the tweets about a specific company on a specific date together with the prices of the company on that date?\n- Are you using the price vectors as value/key and using the text vector as the query?\n- What does 07/06/2021 mean? Based on the context, I guess it means 2021 June 7th?", "summary_of_the_paper": "This paper proposes a co-attention-based model that fusion the representation of tweet and stock price data to make stock trend predictions. The proposed method achieves state-of-the-art performance in a stock prediction dataset.", "summary_of_the_review": "This paper solves the interesting text-based stock prediction problem with a well-motivated method. However, it is hard to identify the novelty and understand the method in detail based on the current manuscripts. I think this paper can be greatly improved when more details and comparisons with existing methods are provided.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RYx3avJcVJ", "reviewer": "Reviewer_GNAc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Comments\nThis paper proposes an approach to fuse text information and stock prices. However,  I have some concerns regarding the current form of the paper. \n1. The technical contribution and novelty are limited as the proposed architecture is composed of several existing techniques.\n2. The paper quality is poor; there are many fundamental grammatical errors and mismatches in the paper, some of which are listed below.\n- In the abstract, the sentence ``Extensive experimentation is performed on tweets and historical price of stock market, our method can achieve a better accuracy compared with the state-of-the-art methods on two evaluation metrics.'' is not grammatically correct.\n- There is a mismatch between figure and paper writings, e.g., the authors mention ConvLSTM in Figure 1, the motivation and the implementation of which are, however not mentioned in this paper.\n- Unclear description of the math formulas, e.g., What is the definition of x in (17)? What is the definition of the operator ⊙ in (20)? What do v_i' and v_m' mean in (18) and (19)? What do z_t and z_T mean in (21) and (22)?\n- The figures should be vector graphics.\n3. Some detailed information is missing, .e.g, there are no details of the used dataset (What is the date range of the datasets? How to split the dataset into training, validation, and test data? [The authors only mentioned they split the data ``by date,'' which is vague.])\n", "summary_of_the_paper": "This paper proposes a method to fuse tweets and stock prices for stock trend prediction flexibly. The authors claim that the proposed method outperforms other existing fusing methods. Furthermore, according to the results of the market trading simulation, this method achieves higher profits than other methods.\n", "summary_of_the_review": "Overall, the quality of this paper in terms of both technical contributions and presentation does not meet the bar of publication of ICLR. (see the above comments)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AQgG3n5rXjp", "reviewer": "Reviewer_K1Z9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength**\n1. It's a meaningful idea to process text and stock prices in parallel and then fuse them.\n2. Using a co-attention transformer to fuse text and stock information is a novel idea that has not been explored.\n\n**Weakness**\n1.  The writings of some parts need more polish\n    * Some references are not in the right places. For example, no reference of the dataset is provided when the dataset is introduced in Sec. 4.1 and no references about temporal attention mechanism are provided in Sec. 3.1.4.\n    * Paragraph 2 of Section 2 can be dived into 2 paragraphs with the second one focusing on fusion technology.\n    * Section 3.1 is not so clear to me. What does endpoint mean? Is it the last day of input data? Does 03/07/2021 refer to March 7th, 2021? What this date and 07/06/2021 the endpoints for 08/06/2021?\n2. It's meaningful to use Bi-LSTM and self-attention to model the text. But why not take one step further to apply Transformer, which is a more powerful model, to encode the text?\n3. The intermediate results of the proposed model are not very clear, e.g. what are the input and output dimensions of each component?\n4. The motivations of some model components are not well explained. Why an LSTM is needed after the co-attention transformer, is it used to aggregate the feature sequence? Why is temporal attention needed afterward?\n5. More details are needed for the market trading simulation experiments for other researchers to replicate it, eg. what's the time period used in it, why are these 12 stocks selected, can we apply this simulation on all stocks, and get some quantitative results?\n6. Does the stock price input data only go through a normalization? If so, it looks the model just directly uses the raw stock price data instead of doing the claimed parallel processing. Besides, what's the dimension of the input and output? More model details would be very helpful.\n", "summary_of_the_paper": "This paper proposed a novel approach to jointly model text and stock price information and fuse them for stock market forecasting. It encodes text and stock price information in parallel and then fuses them using a co-attention transformer. Empirical results over a real-world dataset and trading simulations demonstrate that the proposed approach can outperform the existing baselines.", "summary_of_the_review": "This paper proposed a novel approach to jointly model text and stock price information and fuse them for stock market forecasting. While the proposed approach is solid, this paper still needs more work on its writing and experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a new approach to jointly model text and stock price information and fuse them for stock market forecasting. It encodes text and stock price information in parallel and then fuses them using a co-attention transformer. According to the reviewers, the design of the model is not very well justified and seems to be a little ad hoc. The authors spent quite a few pages introducing background knowledge and the novelty of the proposed model is not sufficiently described. Some details in the experiments are missing, and it is not clear whether the results could be easily reproduced.  There are many writing issues too. As a result, we do not think the paper is ready for publication at ICLR in its current form. BTW, after the reviewers posted their comments, the authors did not submit their rebuttals.
Decision: Reject