Exploring Commodity and Stock Volatility using Topic Modeling on Historical News Articles : Application to Crude Oil Prices

Thorough commodity or stock analysis is a time consuming process because it involves extracting and summarizing information from large amount of data. The majority of financial studies only focuses on numerical data because it is easier to automate using statistical methods and because they represent concrete unchanging values. However, focusing solely on numerical data ignores the context and circumstances of events that led to those values. This study presents a semi-automated workflow for studying stock and commodity prices that incorporates numerical and text based data. The workflow uses topic modeling (Latent Dirichlet Allocation algorithm) to extract important topics from historical news articles. Initial testing of the algorithm was done on 765 articles from PetroWiki. Topic interpretation is made by the user incorporating general knowledge and a simple investigation of representative articles. Timedependent frequency of each topic is calculated and correlated with the price. The workflow is tested on 28,415 New York Times articles mentioning “oil prices” between 1970 and 2015. As expected, oil prices are controlled by a number of inter-connected factors related to politics and financial events. Ten topics were extracted: corporate finance, commodity and US currency, world economy, US Energy policy, emerging economies, Middle East conflict, OPEC, stock market, US elections and politics, and world-OAPEC relations. Time correlation plots show that, in the long run, some topics such as US currency and US Energy policy show a consistent correlation with oil price while others, such as OPEC, change with time. In addition, investigation into a larger number of randomly sampled articles revealed some interesting correlations such as the correlation between oil price and unemployment. Whether the correlation is a causational relationship is unknown. In general, the workflow is useful in identifying important factors that affect the price of a commodity or a stock, aiding the investigator in financial analysis. Introduction Studying the relationship between stock or commodity prices and external factors is an essential component of the investment decision making process. Knowledge of the critical external factors and how the relevance of each factor changes over time is important. For example, a company in the United States that utilizes locally made raw materials in its manufacturing plant will not be susceptible to political and economic changes in China. Relocation of its manufacturing plant to China will however, greatly increase its susceptibility to China. Unfortunately, studying this susceptibility is a time consuming process as it requires going through a large number of reports and financial data. This is especially true for obscure stocks as the investigator does not know in advance the general factors affect the price. In this study, we present a workflow for exploring factors related to stock or commodity prices and study their correlation using machine learning techniques applied on historical newspaper articles. We use the volatility of crude oil price as a case study in this paper because it is of great interest to consumers, companies, and governments. Given the importance of the oil price to global landscape, the observations from this study may shed some lights on hidden factors which should be considered while making financial and political decisions related to the energy industry. Topic Modeling and Latent Dirichlet Allocation The most commonly used method for topic modeling, or topic discovery from a large number of documents, is Latent Dirichlet Allocation (LDA), first introduced by Blei et al. (2003). Simply speaking, the LDA algorithm assumes that each document, as a bagof-words, is a mixture of different topics, and each topic is a mixture of different words. More formally, the smoothed LDA we used assumes the generative process shown in Figure 1Figure 2. There are four steps in this generative process. (a) φ, probability of words in K topics, is drawn from Dirichlet distribution with β as prior weight of words; (b) θ, distribution of topics in each document, is drawn from Dirichlet distribution with α; (c) Z, identity of topic of each word in each document, is drawn from the categorical distribution with probability θ; (d) W, identity of each word in each document, is drawn from the categorical distribution with probability φ, given the topic identity specified by Z. Given W, the collection of documents, LDA uses a variational EM algorithm to estimate the latent variables φ (word distribution for each topic), and θ (topic distribution in each document), with α and β as hyperparameters. For example, we ran LDA for 10 topics on 765 articles from PetroWiki (PetroWiki, 2015). The results, including interpretation of 10 topics, word-topic relationship, and topicdocument relationship, are shown in Figure 2. With the topics neatly interpreted, one can have a quick and high level overview of all the content inside PetroWiki.