A Collaborative Attention Adaptive Network for Financial Market Forecasting

1 INTRODUCTION . In recent years , the use of social media information to predict the financial market has attracted the attention of more and more researchers , and some satisfactory experimental results have been achieved . This is due to the fact that social media information contains investor-related attitudes and subjective sentiments to the financial market , resulting in many investment banks and hedge funds trying to dig out valuable information from media information to help better predict financial markets , which play a key role in predicting the market . At the same time , the efficient market hypothesis ( EMH ) introduced by Fama ( 1970 ) points out that the current price of the asset reflects all the prior information that is immediately available . Therefore , using social media information and the actual price of the current financial market seems to be able to complete the market forecasting more accurately . To the best of our knowledge , the different types of data fusion methods commonly used in the field of financial market forecasting are mainly concatenation or weighted sums . They are simple , direct , and almost no parameter exchange methods to achieve data fusion ( Xu & Cohen , 2018 ; Perez-Rua et al. , 2019 ; Xu et al. , 2019 ) . In the field of computer vision ( CV ) , multimodal fusion methods are mainly used for the fusion of data based on text and images . By realizing the interaction of different information , extracting rich feature information , and obtaining satisfactory experimental results ( Baltrusaitis et al. , 2019 ) . It can be seen that the multimodal fusion method in the CV field seems to be more conducive to feature exchange and feature supplementation than the current method in the financial field ( Lu et al. , 2018 ; Kim et al. , 2017 ) . Therefore , we can infer that choosing an adaptive financial data fusion algorithm to achieve the integration of social media information and actual market prices is effective for improving the forecasting effect of the financial market . However , the current financial market forecasting field basically does not consider too many modal fusion methods , which may be one of the reasons that limit the current forecasting effect . In addition , the role of text and historical prices in the fusion method is different from multimodal fusion tasks , such as face recognition , medical aided diagnosis , and visual analysis . The fundamental reason is that the application scenarios are inconsistent , the source data for other tasks is objective , and the financial data is affected by the sentiment of market investors . Taking into account the difference between tweets and prices , we should pay attention to the effects of both when choosing a fusion method , and use the quantitative analysis of experimental results . To effectively solve the above problems , we propose a collaborative attention ( co-attention ) Transformer approach adaptive to financial market forecasting called CAFF , partially inspired by the recent proposed multimodal fusion in bidirectional encoder representation from Transformers ( BERT ) , which was originally developed to utilize Transformer to realize the feature interaction between text and image data ( Lu et al. , 2019 ) . To verify the rationality of the CAFF method , we included the existing traditional multimodal fusion methods in the experiments for comparison . In summary , the contributions of our work are as follows : • To the best of our knowledge , this work is among the first to introduce a co-attention Transformer fusion approach adaptive to financial market forecasting , which is inspired by the multimodal fusion method and takes into account the characteristics of financial data . • We propose a novel financial market forecasting framework based on the idea of deep fusion to model multisource data analysis . The components of the framework work together to extract tweets and price features in parallel , fuse the features and realize accurate prediction . • Experimental results on stock market forecasting tasks demonstrate that our proposed method achieves a substantial gain over state-of-the-art methods . Moreover , the results also reveal that under the CAFF fusion framework , the quantitative analysis results show that social media information has played a relatively more critical role . 2 RELATED WORK . Some researchers have adopted the method of fusing real market prices and textual information reflecting investor sentiment from social media . In this section , we review the work related to financial market forecasting that fuse prices and text . A variety of analysis approaches based on text and prices to market forecasting have been proposed in the research literature . For example , Xu & Cohen ( 2018 ) proposed a new depth generation model for the stock market forecasting task , which combines text and price signals as the source information . Similarly , Li et al . ( 2020 ) also used the concatenation method to integrate the sentiment information contained in the news data and stock prices to predict the Hong Kong stock market . In addition , Xu et al . ( 2020 ) proposed a stock movement prediction network based on tweet and stock prices by means of incorporative attention mechanism that combines local and context attention mechanisms through incorporative attention to clean up context embedding using local semantics . The approach also makes use of concatenation . It can be seen that the common fusion methods in the field of financial market prediction are still surface feature fusion without parameter interaction . At present , the mainstream fusion technology in the field of CV has attracted our attention , such as bilinear pooling , attention-based fusion , in which bilinear pooling creates a joint representation space by calculating the outer product to facilitate the multiplicative interaction between all elements in the two vectors , which is obviously not suitable for describing text and representing the value of price changes . Lu et al . ( 2019 ) proposed the visual language BERT ( ViLBERT ) model , which extends the Transformer to the fusion operation of processing images and texts . These extracted features can interact through a parallel attention layer , which provides a creative way for feature optimization , but we have to consider the differences between financial data and CV , speech task and other fields . Therefore , developing a method suitable for financial data with the help of the existing fusion methods is a problem worthy of discussion . In summary , using an improved deep learning framework based on a fusion algorithm to analyze financial data to achieve financial market forecasting appears to be effective , which can be confirmed via quantitative analysis . On the basis of the above advanced theoretical analysis , our research motivations are as follows . First , we believe that a framework that processes text and prices in parallel and fully fuses them would perform better than a single data source , where the fusion algorithm plays a key role . However , referring to the existing fusion methods in the CV field , the most natural and effective way to process prices may be to consider the original form as 1-D data rather than as a 2-D matrix image Gupta et al . ( 2020 ) . Moreover , the role of text used to describe the movement of financial market is inconsistent with prices , which are different from the text and image in the crossmodal task of CV fields . The above problems require an applicable prediction framework based on the fusion method to realize the feature extraction and fusion of tweets and prices with temporality , as well as a matching activation function . Hence , we propose a new framework called CAFF , which uses the Transformer model adapted to financial data and a variety of attention mechanisms to collaboratively predict the movement on the target trading day . 3 MODEL OVERVIEW . We propose CAFF , which conforms to the characteristics of financial data , and the complete framework is shown in Figure 1 . The upper of the model deeply extracts text features to learn the text representation of tweets , and the lowers extracts market price features in parallel . The operation with Transformer as the main structure in the middle is used to fuse tweets and price information , and the right side is used to process the merged features and realize the prediction of market trends on the target trading day . 3.1 THE PROPOSED MODEL : CAFF . Considering that there is a certain lag in the target trading day d , we are allowed to simulate and predict other trading days close to d in principle . We can not only predict the trend of the target trading day itself , but also the trend of other trading days during the lag period . For example , if we choose 08/06/2021 as the target trading day , then 03/07/2021 and 07/06/2021 represent the endpoints of the lag period ( the lag period is usually 5 days ) ; thus , we capture the relationships between the predictions mainly within this sample range . However , considering the actual conditions of the financial market , we neglected nontrading days in the calculation process to achieve the effect of effectively organizing and utilizing input data . In general , we can predict a series of movements z = [ z1 , z2 , . . . , zT ] , where the target trading day is zT and the rest are auxiliaries . 3.1.1 INPUT REPRESENTATION . Next , we introduce the processing of text and stock prices to obtain the input representation . Text data is regarded as a feature extraction task in NLP domain . Considering the inherent correlation and volatility , we adopt a relatively direct method to deal with stock prices to maintain the original structure . Tweet-level Model : Generally , tweets posted on social platforms for a given stock often contain multiple inconsistent items on a given trading day . To learn more profitable and valuable information from a large number of investor tweets , we adopt the Bi-LSTM and self-attention to obtain the representation as the input of the fusion component . For each sentence ti , we adopt the pretrained word embedding ( GloVe ) to project each word tag onto the d− dimensional space and the pretrained word embedding as the input of Bi-LSTM . ft = σ ( Wf · [ ht−1 , xt ] + bf ) ( 1 ) it = σ ( Wi · [ ht−1 , xt ] + bi ) ( 2 ) C̃t = tanh ( WC · [ ht−1 , xt ] + bC ) ( 3 ) Ct = ft ∗ Ct−1 + it ∗ C̃t ( 4 ) Ot = σ ( WO · [ ht−1 , xt ] + bO ) ( 5 ) ht = Ot ∗ tanh ( Ct ) ( 6 ) h = [ −→ ht , ←− ht ] ( 7 ) In accordance with the nature of time series data , we use an RNN with Bi-LSTM units and a selfattention mechanism to recursively extract features . The design characteristics of LSTM allow it to model text data ( e.g. , tweet data , news data ) to extract text features and capture the context relationship . Bi-LSTM adds a process of forward and backward concatenation of hidden vectors on the basis of the realization of LSTM . Formulas ( 1–7 ) describe the details of Bi-LSTM . The essence of the attention function can be described as a query ( q ) to a series of key-value ( k-v ) mappings . In current research , the k and v are often the same , that is , key = value . Formulas ( 8– 10 ) describe the details of the attention operations . The only difference between the self-attention mechanism and the abovementioned conventional attention mechanism is that q = k. Self-attention mechanisms have become a hot topic in recent research and have been explored in different tasks . The main reason we choose the self-attention mechanism after Bi-LSTM is that the self-attention is calculated for each word and all words , so regardless of the distance between them , long-distance dependency relationships can be captured to achieve the contextual interaction of tweets to obtain key text features . αi = softmax ( F ( keyi , q ) ) ( 8 ) att ( ( K , V ) , q ) = σNi=1αiXi ( 9 ) attention ( ( K , V ) , Q ) = att ( ( K , V ) , q1 ) ⊕ · · ·⊕ att ( ( K , V ) , qM ) ( 10 ) Price-level Model : Stock prices more intuitively reflect the real market conditions , while tweets contain objective investor attitudes , which allows the two to complement each other . However , the movement of stock has random volatility and is determined by continuous changes in prices rather than the absolute values of the opening and closing prices . Thus , the original stock price vector of trading day d is not fed directly into the neural network ; we employ a normalization strategy to obtain an adjusted closing price . The price adjustment formula is shown below : pd = [ p c d , p h d , p l d ] ( 11 ) pa = pd pcd−1 − 1 ( 12 ) where pcd , p h d and p l d denote the closing price , highest price and lowest price vectors , respectively .

Summary: This paper proposed a novel approach to jointly model text and stock price information and fuse them for stock market forecasting. It encodes text and stock price information in parallel and then fuses them using a co-attention transformer. Empirical results over a real-world dataset and trading simulations demonstrate that the proposed approach can outperform the existing baselines.