On the Robustness of Sentiment Analysis for Stock Price Forecasting

1 INTRODUCTION . Research on the vulnerability of machine learning ( ML ) to adversarial examples ( Biggio et al. , 2013 ; Szegedy et al. , 2013 ) focused , with few exceptions ( Kurakin et al. , 2016 ; Brown et al. , 2017 ) , on adversaries with immediate control over the inputs to an ML model . Yet , ML systems are often applied on large corpora of data collected from sources only partially under the control of adversaries . Recent advances in language modelling ( Devlin et al. , 2019 ; Brown et al. , 2020 ) illustrate this well : they rely on training large architectures on unstructured corpora of text crawled from the Internet . This raises a natural question : when the provenance of train or test inputs to ML systems is ill defined , does this advantage model developers or adversaries ? Here , by provenance we refer to a detailed history of the flow of information into a computer system ( Muniswamy-Reddy et al. ) . We study the example of such an ML system for stock price forecasting . In this application , ML predictions can both serve as inputs to algorithmic trading or to assist human traders . We choose the example of stock price forecasting because it involves several structured applications of ML , including sentiment analysis over a spectrum of public information sources ( e.g. , news , Twitter , etc . ) with little provenance guarantees . There is also a long history of leveraging knowledge inaccessible to all market participants to gain an edge in predicting the prices of securities . Thales used his knowledge of astronomy to corner the market in olive-oil presses and generate a profit . We first reproduce an ML pipeline for stock price prediction , inspired by practices common in the industry . We note that choosing the right time scale is of paramount importance . ML is better suited for low frequency intra-day and weekly trading than high-frequency trading , because the latter requires decision speeds much faster than achievable by ML hardware accelerators . Although there has been prior work on attacking ML for high-frequency trading ( Goldblum et al. , 2020 ) , their experimental setting is 7 orders of magnitude slower than NASDAQ timestamps ( NASDAQ ; 2020 ) , which high-frequency trading firms use . In contrast , ML in low-frequency trading has attracted greater practical interest of the industry , with two major finance data vendors vastly expanding their sentiment data API offerings in the past decade ( Bloomberg , 2017 ; Reuters , 2014 ) . This is to serve the growing demand from institutional and advanced retail market players to use ML models for sentiment analysis at lower frequencies . We adopt this low frequency setting , and collect our data from Twitter and Yahoo finance , which provides 1-minute frequency stock price data . Using these services , we collected tweets related to Tesla and Tesla ’ s stock prices over 3.5 years . We then used FinBERT ( Araci , 2019 ) , a financial sentiment classifier to extract sentiment-based features from every tweet . We describe these methods in more depth in Section 3.1 ; they are general and applicable to any company of interest . We use sentiment features as our input and the change in price as our target to train multiple probabilistic forecasting models . We show that including sentiment features extracted from Twitter can reduce the mean absolute error of a model that only learns from historical stock prices by half . Section 3.2 introduces the probabilistic models used in our work . Predicting stock prices is a known hard task . Even a limited , per trade edge can lead to a large gain when scaled by the number of trades preformed ( Laughlin , 2014 ) . Moreover , typically sentiment analysis is only one of many indicators used in a trade decision . Hence our model only needs to provide a slight non-trivial advantage over random baselines to be effective in practice . We used our forecasts to implement portfolio strategies with positive returns , and measure its performance across other metrics to showcase its advantage . Equipped with this realistic ML pipeline for stock price forecasting , which takes input data from a source with little provenance guarantees ( i.e. , Twitter ) , we set out to study its robustness . Unlike previous settings of adversarial examples like vision , where the realistic incentives for an adversary to change the classification is often unclear ( Gilmer et al. , 2018 ) , in our setting there are clear financial interests at stake . Furthermore , Twitter is already subject to vast disinformation campaigns ( Zannettou et al. , 2019 ) . These make it even more complicated to assess the provenance of data analyzed by ML pipelines . To investigate the robustness of our stock price prediction pipeline , we use adversarial examples to show that our financial sentiment analysis model is brittle in this setting . While attacks against NLP models are not new in research settings , our work demonstrates the practical impacts that attacks on an NLP component ( i.e. , sentiment analysis ) of a system can have on downstream tasks like stock price prediction . We show that an adversary can significantly manipulate the forecasted distribution ‘ s parameters in a targeted direction , while perturbing the sentiment features minimally at test time only . An adversary would determine a parameter and direction , such as increasing variance of forecasted stock prices , and compute a corresponding perturbation . If given control over training data , the adversary ’ s capabilities would only further increase . The contributions of this paper are the following : • We propose a realistic setting for adversarial examples through the task of sentiment analysis for stock price forecasting . We develop a dataset collection pipeline using ticker data and Twitter discussion , for any ticker ( i.e. , company ) . This includes querying Twitter , processing data into a format fit for training , and a suitable sentiment analysis model . • We implement different sentiment-based probabilistic forecasting models that are able to perform better then a naive approach in forecasting stock price changes given Twitter discussion surrounding the company . DeepAR-G , a Gaussian probabilistic forecasting model , outperformed all other models in our experiments . • We subject our pipeline to adversarial manipulation , leveraging information from our model to minimally modify its inputs into adversarial examples , while achieving important changes in our model ’ s output such as shifting distribution parameters of our probability distribution in any direction we wish . We intend to release our code and data should the manuscript be accepted . Beyond the implications of our findings in settings where an adversary is present , we stress that capturing model performance in the worst case setting is important for the domain of finance . These threats are very real : past market anomalies led to the collapse of the Knight Capital Group , and there were legal matters following the suspected manipulation of Tesla stock via Twitter . Therefore , it is important for financial institutions to understand how their ML systems could behave in worst-case settings , lest market anomalies impact these systems in unprecedented ways . Furthermore , unforeseen catastrophic events ( e.g. , a natural disaster or pandemic ) are often hard to model via standard testing procedures be it order generation simulators or backtesting , i.e . simulating trading on replayed past data . Our methodology based on adversarial examples enables institutional traders to assess maximum loss risk effectively . 2 BACKGROUND . Stock Price Forecasting and Market Anomalies . Market agents have long attempted to forecast stock prices . Nonetheless , predicting market behaviour is notoriously difficult for several reasons . If predictive factors become known , they alter the market ’ s behaviour as agents attempt to exploit then . The limit of this behaviour is the efficient market hypothesis ( French , 1970 ) , which states that any information is immediately incorporated in prices , leaving a white noise process . The efficient market hypotheis has generated debate about whether it is possible to systematically outperform the market . See Cornell ( 2020 ) for a recent analysis suggesting that outperformance is possible , and may be linked to outsiders not understanding how such performance is attained . Lastly , markets follow the Red queen effect ( Van Valen , 1973 ) : as parties adapt their prediction capabilities , markets also evolve by becoming more challenging to forecast . The deviations from the efficient market hypothesis are known as market anomalies . They occasionally allow individuals to capitalize off of such discrepancies . See Bass ( 1999 ) for a readable account of a successful application of chaos theory to market prediction . Beyond traditional techniques like arbitrage and standard data analysis , hopes for discovering factors that have not yet been traded away are presently centred on ML techniques , primarily , as the medium by which advances in computing power will drop the costs of research , including , for non-linear predictive factors . Thus , ML is a good fit to tackle the problem stated in the introduction . Indeed , ML models for tasks involving sequential data are now able to learn complex non-linear relationships . Initial work applied these advances to financial time series tasks ( Fischer & Krauss , 2018 ) . Sentiment Analysis in Finance . Sentiment analysis serves as an input signal to trading decisions because it allows one to rapidly digest new information relevant to the market . This requires processing large quantities of text : e.g. , news , financial statements , etc . Previous methods of extracting sentiment and moods were through curated lexicons where every word has a score associated with each sentiment ( Bollen et al. , 2011 ) . Yet , word counting methods poorly reflect semantics since they are unable to factor in features like word order . Bollen et al . ( 2011 ) also received criticism for selecting a small and well specified testing set to achieve good results . On the other hand , training more complex models such as neural networks requires larger datasets . We follow the approach of Araci ( 2019 ) and turn to language models such as BERT ( Devlin et al. , 2019 ) or GTP-2 ( Radford et al. , 2019 ) finetuned on financial sentiment analysis datasets . Such models are able to factor in features that lexicon-based methods could not , achieving a deeper understanding of the text . Time Series Forecasting . The general question of how to predict potential future trajectories from a series of observations is studied in the field of time series forecasting . While traditional modelling approaches like ARIMA ( Box & Jenkins , 1968 ) and exponential smoothing ( Hyndman et al. , 2008 ) have focused on generating forecasts for individual time series , which are typically called local models , more recent techniques based on deep learning enable us to extract patterns from multiple ( potentially related ) time series jointly and capture these patterns in a global model . Popular architectures for time series forecasting include recurrent neural networks ( RNNs ) ( Salinas et al. , 2020 ; Smyl , 2020 ) , convolutional neural networks ( CNNs ) ( Oord et al. , 2016 ; Borovykh et al. , 2017 ; Bai et al. , 2018 ) with 1-dimensional convolutions over time , as well as transformers ( Vaswani et al. , 2017 ; Lim et al. , 2019 ; Li et al. , 2019 ) using attention-based mechanisms . More details about the exact forecasting framework used in this work are established in Section 3.2 . Adversarial Examples . Extensive prior work in adversarial examples is in the vision domain ( Szegedy et al. , 2013 ; Su et al. , 2019 ) . Despite the fundamental advances in our understanding of the robustness of ML these works enabled , the practicality of such work is limited , and the incentives of the adversary to achieve such forms of misclassification is unclear ( Gilmer et al. , 2018 ) . Recent work has focused on more practical domains , such as malware analysis ( Kolosnjaji et al. , 2018 ) or sentiment analysis of movie reviews ( Samanta & Mehta , 2017 ) . Studying the implications of adversarial examples in financial applications is clear : the financial gain will motivate malicious individuals to attack ML models deployed by financial institutions . This begs the question of whether these applications of machine learning to finance will exhibit the same lack of robustness than their counterparts did in the vision domain . In our work , we adapt the common strategy of determining the model ’ s sensitivity to perturbations in the input through an analysis of the model ’ s gradients . In particular , we leverage previous work on RNNs ( Papernot et al. , 2016 ) .

Summary: In this paper, the authors studied the problem of adversarial ML in stock price forecasting. They first replicated an industry standard pipeline, which performs a sentiment analysis of Twitter data to forecast trends in stock prices. Then, they show that an adversary can exploit the lack of provenance to indirectly use tweets to manipulate the model’s perceived sentiment about a target company and in turn force the model to forecast price erroneously.