Exploring the Forecasting Potential of Company Annual Reports

Previous research indicates that the narration disclosure in company annual reports can be used to assist in assessing the company's short-term financial prospects. However, not much effort has been made to systematically and automatically assess the predictive potential of such reports using text classification, information retrieval, and machine learning techniques. In this study, we built SVM-based predictive models with different feature selection methods from ten years of annual reports of 30 companies. We used feature selection methods to reduce the term space and studied the class-related vocabulary. Evaluation of predictive accuracy is performed with cross validation and t-test significance tests. We compare different models' performance and analyze misclassification rates by year and by industry. We identify the strengths and weaknesses of each model. Our results support the feasibility of automatically predicting next-year company financial performance from the current year's report. We suggest text features can be further studied to understand their roles as indicators of company's future performance. This research paves the way for large-scale automatic analysis of the relationship between annual reports and short-term performance, as well as the identification of interesting signals within annual reports.  Company annual reports (10K filings) are freely available to the public and contain required disclosures, quantitative summaries of the company's financial performance as well as textual discussions. These reports are of great importance in helping investors, corporate managers, and financial analysts with their decision-making. Studies have shown that the narration sections of 10K filings provide information that is as useful as the financial ratios to financial analysts while predicting the company's future prospects (Roger & Grant,1997, Schipper, 1991). The SEC (Securities and Exchange Commission) also requires the reporting of the firm's strategies and managerial priorities, and its view of the past year's performance and future prospects. The major mandatory disclosures in annual reports include reasons for price and sales changes, reasons for revenue and cost changes, planned expenditures, known trends, and future liquidity positions. Annual reports have been studied as a marketing and communication tool that the corporation uses to convey an image or messages to its stakeholders (Herreman & Ryans, 1995). More recent studies on the relationship between the reports and firm performance have focused on special sections of the reports, such as the chairman's statement (Smith & Taffler, 2000), management discussion and analysis (MD&A) (Bryan, 1997), president's letter (Abrahamson & Amir, 1996) as well as the general writing style and readability (Subramanian et al., 1993). The methods these studies employ are generally semi-automatic, including content analysis, readability measurements, manual annotation and categorization, linear discriminant analysis, logit model and other statistical analysis. The main contributions of these studies are that the researchers were able to identify special features of the writing in general, or special disclosure variables, that correlate with certain performance ratio or general profitability. For example, Subramanian et al. (1993) found that good performers used strong writing in their reports while poor performers' reports contained significantly more jargon or modifiers and were hard to read. Smith & Taffler (2000) identified thematic keywords from chairman's statements and generated discriminant functions to predict company failure. Bryan (1997) showed that the discussion of future operations and planned capital expenditures were associated with one-period-ahead changes in sales, earnings per share, and capital expenditure. Kohut & Segars (1997) studied president's letters in annual reports and suggested that poor performing firms tended to emphasize future opportunities over poor past financial performance as a communication strategy. These studies emanate from the intuitive recognition of a link between the textual report content and corporate performance. Their findings suggest that combining the textual analysis of the reports with the quantitative data in the financial statement can assist the prediction of company performance and even failure and bankruptcy. Predictive models for financial performance have been studied mainly using the financial data and various machine-learning techniques such as classification (Turetken, 2004). The goal is generally to identify prominent financial ratios with good predictive power, or better performing classification algorithms such as neural networks. Surprisingly, little research has been done on utilizing the textual content of annual reports to build predictive models, despite findings that the reports have the potential to serve as indicators of company future prospects. The most related work in this direction is that of Kloptchenko et al. (2002) and Visa et al. (2000). In Kloptchenko et al. (2002), company quarterly reports and corresponding financial ratios were clustered with prototype matching clustering and SOM clustering respectively. Although the two clusters did not coincide, the authors found that changes in textual reports tended to occur ahead of the changes in financial performance. The wealth of information in these reports, especially the narration (textual) portions acknowledged as important for human analysts, remains untapped for machine learning applications. Set in this background literature, we see a well-justified opportunity to explore methods for predicting company financial performance from annual reports. In this research, we explore the feasibility of using the textual content of annual reports for a given year to predict the company's financial performance in the next year. We measure financial performance as return on equity (ROE) ratio. We applied the bag-of-words vector representation to represent each annual report, and performed Support Vector Machine (SVM) based classification with crossvalidation to evaluate the predictive accuracy. We also experimented with different feature selection methods to reduce the term space and examine the vocabulary subset capable of predicting a certain performance class. The goal of our study is to establish a baseline for building predictive models from the textual content of annual reports and to analyze different models' strengths and weaknesses in this application domain. More specifically, we address the following research problems: • Determine the feasibility of building predictive models from annual reports measured by classification accuracy • Evaluate different predictive models' strengths and weaknesses in predicting the specific class of future financial performance • Examine the potential of detecting interesting textual (vocabulary) features from annual reports that may serve as signals of future financial performance • Detect patterns that may exist in different industries and different years The key contribution of our research is the application of text classification methods in the financial domain for knowledge discovery. We attempt to build and study different classification models to better capture the predictive signals in company annual reports, if they exist. Our analysis also explores the trade-off between different models and the challenges faced when building such applications. The rest of the paper is organized as follows: In section 2, we will present our methodology related to data collection, prediction problem modeling, selection of modeling approach, and evaluation methods. In section 3, we will present our experimental results in terms of predictive accuracy. We will compare different models to address the research goals presented above. In section 4, we will present our general observations and outline our plans for further study.  Data Collection and Class Definition In this study, we first had several domain experts (two professors and a Ph.D. candidate in Accounting) help us identify our data. With their help we selected a total of 30 companies from 3 industries (pharmacology, IT, and banking). Each company has at least 10 years of consecutive filings with the SEC and has had performance that has fluctuated over the time frame. We retrieved automatically from EdgarScan all the 10K filings of these companies for years 1990 to 2003. Our domain experts also helped us collect the financial measurements for each firm/year from the COMPUSTAT database. We calculated the Return On Equity (ROE) ratio for each firm/year. ROE ratio measures the earnings power of owners' equity. It shows how much income was earned for every dollar invested by owners. The increase and decrease in ROE ratio indicates the change of the firm's earnings power. With the guidance from our domain expert's analysis, the ROE ratios were partitioned into 3 classes. We use t to refer to the year corresponding to the annual report year and t + 1 to refer to the following year. • If the ROE ratio in year t + 1 is within 5% of the ROE ratio in year t, then the company's year t + 1 performance is classified as belonging to a “neutral" class. • If the ROE ratio in year t+1 is greater than the ROE ratio in year t by more than 5%, then the company's year t + 1 performance is classified as belonging to a “positive" class. • If the ROE ratio in year t + 1 is less than the ROE ratio in year t by more than 5%, then the company's year t + 1 performance is classified as belonging to a “negative" class. Our hypothesis is that the reports carry enough signals to predict the next year's performance. Thus we pair the document with the class of the next year's performance. When we retrieved all the 10K filings for these 30 companies, we obtained 316 documents. Among these, 37 documents could not be labeled because of lack of ROE data, resulti