TS-BERT: A fusion model for Pre-trainning Time Series-Text Representations
Authors: Jiahao Qin
Lu Zong
Venue: ICLR 2022 Submitted (2022)
Keywords: Time Series-Text Representations
Pre-training
Mutilmodal

Abstract: There are many tasks to use news text information and stock data to predict the crisis. In the existing research, the two usually play one master and one follower in the prediction task.
Use one of the news text and the stock data as the primary information source for the prediction task and the other as the auxiliary information source.
This paper proposes a fusion model for pre-training time series-Text representations, in which news text and stock data have the same status and are treated as two different modes to describe crises. Our model has achieved the best results in the task of predicting financial crises.

Under review as a conference paper at ICLR 2022
TS-BERT: A fusion model for Pre-training Time Series- Text Representations
Anonymous authors Paper under double-blind review
Abstract
There are many tasks to use news text information and stock data to predict the crisis. In the existing research, the two usually play one master and one follower in the prediction task. Use one of the news text and the stock data the primary information source for the prediction task and the other as the auxiliary information source. This paper proposes a fusion model for pre-training time series-Text representations, in which news text and stock data have the same status and are treated as two different modes describe crises. Our model has achieved the best results in the task of predicting financial crises.
Introduction
Forecasting the stock market trend from multiple angles is of great significance to economics researchers and sociological researchers.
First of all, there are extensive studies in finance to solve the problem of forecasting the stock market. There are two main motives for them. The first is that predicting the stock market trend could reveal the specific factors that drive the stock market trend. Socond, by predicting the stock market trend, it is possible to evaluate the pricing of particular assets Pönkä (2017).
The influence from the stock market would spread to other areas due to the significant impact on the society from financial. How to predict and prevent significant stook volatility crises critical research topic. Economic researchers in academia and the financial industry have never stopped their research in this area. Related research methods are constantly updated.
Financial time-series forecasting is one of the most important research tasks in these fields. Economics researchers are increasingly accepting the use of deep learning for related re- search. More and more tasks use AI technologies such as machine learning and deep learning to predict financial time series. Researchers in related fields have created various models and published large number of papers accordingly. After introducing deep learning models into this field, the results obtained are significantly better than traditional machine learning models Sezer al (2020) For example, the time series forecasting model based on the trausformne Wi et, al (2020a), and time series forecasting model based on graph neural network Wu et al. (20201). Researchers are becoming more and more interested in de- veloping financial time series forecasting models Thakkar & Chaudhari (2021). While the modelling methods are becoming more and more diversified, the complexity of the model is also increasing. Fortunately, these behaviour results are gratilying, and the accuracy of financial time series forecasting tasks is constantly improving.
Not only is the model itself becoming more powerful and more complex, but even the information on which the prediction task itself is based is also becoming more and more diverse. From time-series data itself to macro-financial factors to text data such as news, the data of financial time series forecasting tasks have become more abundant. Especially for text data, text mining has become a significant research field. It has been applied in many fields such as medical treatment, business, finance, education, and many others Elagamy et al. (2018).

---

Under review as a conference paper at ICLR 2022
News content is one of the most critical factors affecting the stock market. Analysis of the impact of news when predicting stock market volatility would lead to more reasonable forecasts. Stocks are one of the most important components of financial markets, and they could directly impact regional and global economies. Precise forecasts of stock trends could help support eritical business and financial decisions and even political decisions.
Therefore, we use stock time series data and news text data to enter the main methods of this article on stock market trends. The primary method of our project is to improve the accuracy of prediction based on the fusion representation of text and stock lime-series data.
2 Related work
The use of data mining techniques to analyze and predict the stock market has been widely recognized Nikfarjam et al, (2010). This method mainly uses structured data, such as past prices, historical returns, or other stock characteristic indicators. The task of using data mining to predict financial time series has achieved many results. Samitas et al. (2020) has established an early warning system, which dramatically improves financial risk early- warning accuracy and could give early warning signals in advance.
The method of text mining is different from that of data mining. It is not easy lo extract relevant information from unstructured data such as news. So this type of method has relatively few applications in financial time series forecasting tasks.
The trend of stocks is entirely random. Using data mining methods to predict the trend of stocks is very dependent on a large amount of data. However, historical data that stretches the time axis too long does not have a high reference value for the current state Patel al. (2015). Furthermore, they believe that it nst casy to provide reasonable explanation ellect simply by using numerical prediction Schumaker et al. (2012). News content is one of the most critical factors affecting the stock. II the news is classified in adyance, and then these tags are passed in when forecasting, better results could be obtained Nikfarjam et al. (2010).
News text data usually contains many events, and these events could be used for the sup- plementary analysis of random behaviour. There is lot of free news and financial data. If both could be used at the same time, it may improve the interpretability of the predictive model. Moreover, they proved that the correct available information could improve the performance of sequence prediction models.
Text information has a very unfriendly feature: their language structure and writing style are very different. It is not easy for machine deep learning technology to learn information and patterns from such unstructured data. This difficulty is mainly reflected in the modelling and training phases. In order to solve these problems, there are two leading solutions: subject terms, modelling, and keyword modelling.
The solution we currently use is keyword modelling, which is used to characterize news text.
Many researchers use news in stock forecasting tasks. For example, the news labels are input into the predictor as a feature of stocks Nikfarjam et al (2010), and Fung et al, (2002) also proposed a system that only uses the news to predict stock trends. A stock trading carly warning system is also established through the classification of news Mittermayer & Knolmayer (2006). News used in such tasks because news has an impact not only on the stock market itself but also on the stock market participants. They discover that specific news articles can influence investors' decision-making Li et al (2014). In addition, they prove that the text information of the news could help investment institutions conduct intraday risk management Groth & Muntermann (2011).
There are many examples of using news in tasks related to stock forecasting, and most of them have achicved better results. However, their mothod is to classify the nows text as a stock feature or only use the news text without using stock data.
In addition to fusing multiple information together, some researchers use different models and different types of information to make different predictions He et al. (2019). Then they
2

---

Under review as a conference paper at ICLR 2022
merge these single models and use model fusion solution to improve the prediction effect. Model fusion could perform various operations to fuse two or more models and multiple specific features. It must be noted that these operations may be different from the possible operations performed by feature fusion. Such operations and optimization could be applied to wide range of models. Extensive experiments have proved that the performance of the fusion model is 3%-5% higher than the performance of a single model.
Fusion models usually require the of information features from multiple sources, and then use the superior performance of different. models for certain types of information to improve the final performanccThakkee Thakkar & Chaudhari (2021).
Since some studies already prove that news text alone could also effectively predict the stock market, we suppose that news could be an independent observational state of the stock market. Using news text and stock price data simultancously and treating them as two modalities, we could use multi-modal lcarning method in the prodiction task.
3 TS-BERT
3.1 The BERT model
BERT Devlin et al. (2019) proposes to learn language representations by using "masked language model" training objective. more detail, let I1,..., 2g be of discrete tokens, E X. We can define a joint probability distribution over this set as follows:
px ()
1 Z(0)
II
dr( 0) exp Σ log oi (x ())
Use BERT to establish a self-supervised key semantic feature extraction method, and achieved the best results. This process avoids the time for manual tagging of keywords and does not require any prior knowledge. This allows us to easily apply this method to huge financial news data sets.
3.2 TS-BERT
Our world experience is multi-modality we see objects, hear sounds, feel textures, smell smells, laste lastes. Modalily refers to the way something happens experiences. When research question contains multiple modalities, has the characteristics multi-modality. In order for artificial intelligence to make progress in understanding the world around us, it needs to interpret these multi-modal signals simultaneously.
For example, images are usually associated with tags and text explanations. The text contains images to more clearly express the article's contral idea. Different modes have vory different statistical properties.
In our task, the volatility of stocks is our goal. The statistical characteristics stock volatility exist in the time series data of stocks and the news text related to stocks.
3

---

Under review as a conference paper at ICLR 2022
newe- 18x1
Sub-Nebwork transformer
WeighLneews led
Network
RNN
Outbut
Figure 1: Full-model Diagram.
3.2.1 Fusion Embedding
The fusion embedding method could be divided into two parts. The first part is to embed the news text, the second part is embed the stock data, and then the third step is fuse the two. In terms of design, this is a multi-modal neural network learning method.
3.2.2 News Embedding
In text embedding, we use keyword extraction technology, which uses BERT to embed to creale keywords and key phrases that most similar to text.
Although there are many methods available for keyword generation, such as Rake, Yake, and TF-IDF, the performance of these once-popular methods is no longer powerful in the current deep learning field. Due to the advent of pre-trained models, it is easy to create more powerful method to extract keywords and key phrascs. This the design basis of our method, using Bert embedding and superficial cosine similarity to find the most similar sub-phrases in the news text to the news itself.
First, use BERT to extract news embedding to obtain news-level embedding. Then use no-gram to extract the embedding of the word. Finally, we use cosine similarity to find the most similar words phrases to the news. Then define the most similar words or phrases as the words and phrases that best describe the exclusive news.
We feed the news to BERT, and obtain the contextual feature vector of each word. The vectors of words in a news is averaged in order to get its news embedding vector. Then embedding vector. The idea is that keyword should capture meaning of news, and thus should be closer to the news embedding. The similarity of the embeddings to the news embedding is obtained using cosine similarity metric.
choose the words close to
4

---

Under review as a conference paper at ICLR 2022
News
Bert
Doc. embedding
N-gram
Word_1 embedding
Word_2 embedding
Word_n embedding
cosine similarity
Top
embedding
Figure 2: News embedding.
The advantage of this method is that we do not need to start training from scratch and adjust the parameters multiple times. The performance of BERT itself has been widely recognized.
3.2.3 Stock time series data Embedding
There are also many modelling methods for stock time-series data. However, because do not need to use very complex model in this task, we choose to use a basic RNN to embed the stock time series data.
Embedding connection Although combining different modalities or information types to improve the effect intuitively an attractive task, how combine different noise levels and conflicts between different modules in practice is a challenge. The most common method in practice connect high-level embeddings of different inputs and then apply softmax.
The problem with this approach is that it would give equal importance to all sub- networks/modde which very unlikely in reality. News and slock prices may not have the same influence on the forecast, so we need to calculate the different weights they play.
3.2.4 Weighted combination of networks
Because the renshio between the two modalities of text features and time-series features is relatively complicated, could not only use simple fusion methods to describe this relationship. And it would cause the fusion of feature vectors to have too high dimensionality.
We use MCB to solve this problem. MCB pools the fused features. First, the vector outer product calculation is performed on the feature vector of each location of the feature map obtained by the convolution. Then sum pooling is performed on the results of the outer product calculation all locations to obtain the feature vector x. x gets the final feature after signed square root and T.2 normalization.

---

Under review as a conference paper at ICLR 2022
The operation of bilinear pooling can be expressed as:
B(X) = Ex,x,
SES
MCB maps the result of the outer product to low-dimensional space, and does not need to explicitly calculate the outer product.
4 EXPERIMENT
4.1 Data
We usod data from the kaggle competition. We carry out some necessary processing of these data. In the end, the data set used for the experiment contained data from total of 300 companies from NYSE and NASDAQ. The period is from 2008 to 2021, with a total of more than 2,800 trading days. Each day's 's data for each company includes the corresponding stock price and news related to the company.
We consider the data of 300 companies We split the data set into two parts: training and testing, 85% is the training set, and 15% the test set. In addition, fall in cycle as crisis and the rest as crisis-free.
whole, our data set length is around 840,000.
regard the top 5% and tail 5% of the stock's daily rise and
4.2 Baseline Methods
For contrastive experiments, we consider baselines with two different methods:GCN classifier and CNN classifier with LSTM embeddings.Sin Since these models do not have multi-modal application cases similar to our text-time series data, this result could not be considered as the best performance of the model on such tasks.
4.2.1 GCN classifier
We use words and stock features as news nodes to construct news with graph structure. Then use TF-IDF to characterize the relationship between words and news. In the training phase, we use fixed-length window to divide the crisis or not as monitoring label for news.
4.2.2 CNN classifier
We use LSTM to extract the key semantics of the text, and then use CNN to extract the key features of the semantics.It should be noted that only news data is used in CNN classifier.
4.3 Results
As shown in tablel, both our model and baseline model have high precision, but fewer false warnings are crucial for crisis prediction tasks, so we also calculated the recall.
Table 1: Result of precision and recall.
Model Precision
Recall
F1 Score
TS-BERT 0.98
0.95
0.96
MTGNN
0.92
0.88
0.90
CNN 0.91
0.76
0.83

---

Under review as a conference paper at ICLR 2022
4.3.1 Ablation experiment
In order to verify that the text and time series data respectively made positive contributions to the experiment, we did an ablation experiment and showed the results in table2
Table 2: Comparison of results between single modal experiment and dual modal experi- ment.
Model
Precision
Recall
F1 Score
TS-BERT news only)
0.84
0.89
0.86
TS-BERT (stock only) TS-BERT
0.94
0.81
0.87
0.98
0.95
0.96
MTGNN
0.92
0.88 0.76
0.90
CNN
0.91
0.83
5 CONCLUSION
Using multi-modal learning methods to improve the accuracy of stock risk volatility predic- tion is one of the main directions of work. In future work, we have additional possible improvements.
At present, we use the top3 self-relevance keywords in the news as news representations, and there may be possible that the impact of these three keywords on the stock market is not the same. We could score the relevance of the news stock classification before extracting the keywords. Then we use the highest score as the stock classification tendency of the news and then extract the top3 words of this classification tendency in the news.
Another possible improvement is that we may design new crisis modelling method, which aims to accurately describe the crisis fluctuations in the stock market and improve the inter- pretability. And then, using more complex and more powerful time-series data embedding method may also improve the performance of our model.
In order to be more in line with the actual stock market, perhaps the weight ratio of precision to recall is not the same as in the standard F1. Therefore, changing the recall weight is also one of the important applications of the F1 score. After the change, the formula of the F1 score could turn to:
Precision Recoll
F3 - -1+ (1)
(32 * Precision) Recall
If necessary, experimental data, model structure, etc., may be further improved.
Acknowledgments
Use unmumbered third level headings for the acknowicdgments. All acknowledgments, in- cluding those to funding agencies, go at the end of the paper.
References
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv:1810.04805 [cs), arXiv: 1810.04805.
May 2019. URL http://arxivvorr/abbee /aabbb///110 1810000405,
Mazen Nabil Elagamy, Clare Stanier, and Bernadette Sharp. Stock market random forest- text mining system mining critical indicators of stock market movements. Tn 2018 2nd

---

Under review as a conference paper at ICLR 2022
International Conference Natural Language and Speech Processing (ICNLSP), pp. 1-8, April 2018. doi: 10.1109/ICNLSP.
Gabriel Pui Cheong Fung, Jeffrey Xu Yu, and Wai Lam. News Sensitive Stock Trend Prodiction. In Ming-Syan Chen, Philip S. Yu, and Bing Liu (eds.), Advances in Knowledge Discovery and Data Mining, Lecture Notes Computer Science, pp. 481 493, Berlin, Heidelberg, 2002. Springer. ISBN 978-3-540-47887-4. doi: 10.1007/3-540-478877
48.
Sven S. Groth and Jan Muntermann. An intraday market risk management approach based on textual analysis. Decision Support 5ystemss_50((446688888888 March AIILISSSN 0167-9236 doi- 10 1016/5 / dss 2010.08.019. URL https:/ http://wwwww.iiecccdeeeeetcc /science/
article/pi/$00066926100000
Qun He, Zhangli Cai, and Wenjing Liu. Prediction of Listed Companies' Revenue Based on Model-Fused. In 2019 International Conference on Machine Learning, Big Data and Busi- ness Intelligence (MLBDBI), pp. 257-260, November 2019. doi: 10.1109/MMB. MLBDDB44998. 2019.00057.
Qing Li, TieJun Wang, Ping Li, Ling Liu, Qixu Gong, and Yuanzhu Chen. The effect of news and public mood stock movements. Information Sciences, 278:826-840, Septemberr 2014 ISSN 0020-0255 doi- 10 1016/j ins 2014.03.096. 2014..0..0 URL https:/ http:://////www.iieenn. /twp://wieneeerrree...
comm/sscienee//tttill//////////0 pS000000000051111111111 40000077995440889.
Marc-andre Mittermayer and Gerhard F. Knolmayer. NewsCATSS: A News Categorization and Trading System. In Sixth International Conference on Data Mining (ICDM'06), pp. 1002-1007, December 2006. doi:
ISSN: 2374-8486.
Azadeh Nikfarjam, Ehsan Emadzadeh, and Saravanan Muthaiyah. Text mining approaches for stock market prediction. In 2010 The 2nd International Conference on Computer and Automation Engineering (ICCAE), volume 4, pp. 256-260, February 2010. doi: 10.1109/
ICCAE.2010.5451705
Jigar Patel, Sahil Shah, Priyank Thakkar, and K Kotecha. Predicting stock market index using fusion of machine learning techniques. Expert Systems with Applications, 42(4):
March 2015. ISSN 0957-4174 doi: 1016//01/.......1.11 http://..esaaa.11.1..... URL https: 2162 March
Harri Pönkä. Predicting the direction of US stock markets using industry returns.
Empirical Economics, (2(44)1451-148) June 2017 ISSN 1435-8921. doi: 10.1007/
500181-016-1098-0. URL.
Aristeidis Samitas, Elias Kampouris, and Dimitris Kenourgios. Machine learning as an early warning system to predict financial crisis. International Review of Financial Analysis, 71:
101507__cttbbrr October 2020. ISSN 1057-5219
doi: 10. 1016/1 wttf///aaaaaaaaaaeeeaaaaaa irfa 2010.101507. URL https:
www.scieneedireet.cmm www.c/ooooocccc///eeeeeeeeee /ttp
Robert P. Schumaker, Yulci Zhang, Chun-Neng Huang, and Hsinchun Chen. Evaluating sentiment financial news articles. Decision Support Systems, 458 June 2012,
ISSN 0167-92366 doi: 10 1016 http:/dsss..222..00.0 2012.03.001. URL
http:///arrriee/p//////677222/ science/article/ pii/SS16722366660111111 200.66772261210077. 2000875
Omer Berat Sezer, Mehmet Ugur Gudelek, and Ahmet Murat Ozbayoglu. Financial time series forecasting with deep learning: : A systematic literature review: 2005 2019. Applied Soft Computing 90-106181 May 2020 ISSN 4566 doi: 10 1016/1 asoc 2020. 2020.106181. URL
Ankit Thakkar and Kinjal Chaudhari. Fusion in stock market prediction: decade survey on the necessity, recent developments, and potential future directions. Information Fusion,
107
2021 ISSN 1566-2535. doi: 10. L016/J_infffs.. 2020.08.019, URL https:
www.ssieencedireetcco wwm.:iirrrettcciicooommmmmmmm wtt.s//eeeedddeee.t.o/e//eeee//e /tt.://e////////50553333333344
Neo Wu, Bradley Green, Xue Ben, and Shawn O'Banion. Deep Transformer Models for
Time Series Forecasting: The Infinenza Prevalence Case. arXiv:2001.082
[cs, stat),,
January 2020a. URL
arXiv: 2001.08317.
8

---

Under review as a conference paper at ICLR 2022
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Connecting the Dots: Multivariate Time Series Forecasting with Graph Neu- ral Networks. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 753-1 763.. Association for Computing Ma- chinery, New York, NY, USA, August 2020b. ISBN 978-1-45033-999-4. URL https: //doi.org/10.1145// /09///00.14//339486666 18833//3/113


Reviews:
[{"review_id": "zuG-zKflBn2", "reviewer": "Reviewer_oxEh", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The method is well-motivated and the problem is interesting.\n- The proposed method achieves the best performance in the empirical study,\n\nWeaknesses:\n- There are too many apparent and fundamental mistakes in this paper:\n    - Though it looks similar, this paper does not follow ICLR's official format. For example, the title of each subsection is in lower case and the citations are in the wrong format.\n    - There are some outstanding drafts at the end of the paper (see Page 7 before the acknowledge section).\n    - The authors did not hide the acknowledge section in this submission, though there is no information leak.\n    - There are grammatical errors in most of the paragraphs.\n- The technical contribution and novelty are relatively limited or not well presented.\n- Minor problems/suggestions:\n    - Please add a link or reference to the Kaggle dataset you use.\n    - What does MCB mean. Please cite the paper which proposes it or briefly explain it.\n    - This picture is so unclear. It would be much better if using a PDF version.\n", "summary_of_the_paper": "This paper introduces a method that combines the representation of text and time series to make financial crisis predictions. The proposed methods outperform all the baseline methods.", "summary_of_the_review": "This paper proposes a new and powerful method to solve an intersting problem. However, the technical contribution is limited and the presentation quality needs to be greatly improved to meet the bar of ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "wIbQVRAFfgV", "reviewer": "Reviewer_YK4B", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points:\n\n- The combination of text and financial data is a worthy area of study (although this work is not the first to try to do this).  Using BERT is a promising approach here.\n\n- An ablation study showed that both the news and stock portions of the model were important to get the best performance.\n\nWeak points:\n\n- The proposed model is not particularly novel, as far as it is explained.  Surely the authors are not the first to combine text and financial data, and the precise novelty is not well argued.\n\n- The fusion aspect of the approach is fairly pedestrian and does not consider how text and stock prices interact and change over time.\n\n- The details of the proposed TS-BERT model are a bit unclear in the paper. Figure 2 is helpful but it does not suffice. There needs to be equations in the paper describing the precise details of the model. I definitely could not replicate the authors' method based on what is included in the paper.\n\n- It is unclear why keyword extraction was performed based on the BERT embeddings, instead of simply using the BERT embeddings.  There would be a loss of information in this step.  If this step is useful, experiments should be provided which show it. Similarly, the role of N-grams in Figure 2 is unclear.\n\n- The training algorithm for the model needs more explanation.  What is the loss function, what is the optimization algorithm, and how is fine-tuning used?\n\n- Only one dataset is used in the experiments.\n\n- There are insufficient baselines.  Other state of the art text + stock price models, and published models considering these aspects separately, should be compared to (e.g. models which were mentioned in the related work section).\n\n- Both of baselines which were used are a bit strange and unclearly explained.  The GCN method represents word and stock features as a graph, but this data is not naturally graph structured, and it is unclear how the graph was constructed or what the rationale was.  The use of TF-IDF in this context is also very unclear.  Similarly, it is not clear why the CNN model also includes an LSTM component, or how these two were put together.  I don't know why there weren't simply separate LSTM and CNN baselines.\n\n- There are numerous grammatical errors throughout the manuscript.  There are other clarity and framing issues as well, e.g. the paper goes back and forth between saying that the task is stock forecasting (i.e. predicting stock price), and predicting the occurrence of crises.\n\nAdditional feedback / minor suggestions:\n\n- The equation regarding BERT doesn't seem to be the right one. Or at best, it does not capture the unique aspects of BERT, such as the transformer-based architecture, bidirectional representations, and the masked language model and next sentence prediction training objectives. \n\n- There needs to be a citation for the MCB method. What does the acronym stand for? Including equations to explain it would help as well, if space allows.\n\n- \"table1\", \"table2\" - should be \"Table 1\", \"Table 2\".\n\n- The Acknowledgments stub from the conference template was left in the paper.\n\n- The title is not consistent in its use of capitalization.\n\n- BERT is not consistently capitalized throughout.\n\n- Section 3.2.3 heading has inconsistent capitalization. Below it, \"Embedding connection\" looks like it should be a heading, instead of the first two words of a different sentence.\n\n- Many citations should be put inside parentheses, especially when they occur at the end of sentences. E.g. \"Sezer et al. (2020)\" should be \"(Sezer et al., 2020)\" in its use on page 1. Use the Natbib \\citep{} command.\n\n- Is MTGNN the method which is referred to as GCN in Section 4.2? Name it consistently.", "summary_of_the_paper": "The paper proposes a method for predicting stock market crises using a deep learning approach which combines time series stock market data with text from news articles.  Experiments find that the method works better than the same model using only news or only stock prices, and a couple of deep learning baselines.", "summary_of_the_review": "While the combination of text and financial stock data via BERT is an interesting task, the paper presently has multiple serious issues in novelty, clarity, and experimental rigor.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OSmpoAaRlSw", "reviewer": "Reviewer_7P8j", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Reasons to accept:\n\n-\n\nReasons to reject:\n\nI appreciate the authors’ effort. I am certain that they have made a great effort to develop the model and prepare the paper. I’d like to also remind that my task, as a reviewer, is to provide a helpful review so that the authors can receive an outsider view.\n\n- In my opinion this article is more like a blog post which is *badly* formatted in a scientific paper. More specifically: The language is below standard, the related work coverage is very limited, the claims are contradictory, the model is too naive, the experiments are insufficient, the baselines are weak, and the analysis is superficial.\n\n- The paper has no contribution in my opinion. The main idea of the paper has been proposed before [1], and is not cited in the paper.\n\n- The authors neither in the introduction nor in the related work mention their actual contribution compared to existing works. The reader should infer this himself.\n\n- There are a number of contradictory claims in the paper. The authors state:\n“The trend of stocks is entirely random.”\nIf it is random, then why would you make an attempt to predict it?\nAgain the authors state that their baselines are weak:\n“Since these models do not have multi-modal application cases similar to our text-time series data, this result could not be considered as the best performance of the model on such tasks.”\nIf they are weak, why you didn’t use better baselines?\n\n- BERT already has a document vector representation called [CLS]. There is no need to take average of words.\n\n\n[1] Fune et al., Stock prediction: Integrating text mining approach using real-time news. CIFEr 2003.", "summary_of_the_paper": "The authors propose a model for stock prediction by aggregating the embeddings of automatically mined news keywords and the embeddings of stock series. The authors evaluate their model in a large dataset and show that it outperforms a TF-IDF based model and a CNN based model.", "summary_of_the_review": "The presentation of the paper is below standard, it has no contributions, and the experimental setup is not convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for predicting stock market crises using a deep learning approach which combines time series stock market data with text from news articles. Their experiments show that the proposed method works better than the same model using only news or only stock price data, and a couple of deep learning baselines. All the reviewers pointed out that this paper is lack of novelty and significant technical contributions. The experiments are performed on a single dataset with incomplete baselines, and hence insufficient to support the claimed advantages of the proposed method. The writing quality is not up to the standards of an ICLR papers, with too many grammatical mistakes, typos, and unjustified arguments/claims.  The clarity of the writing is poor.

The authors did not provide their rebuttal.
Decision: Reject