Using NLP on a company’s annual reports to predict near bankruptcy | by Gary Licht | Towards Data Science

After trading fixed income and currencies for many years, I was looking for a project as an introduction to natural language processing (NLP). After witnessing so many scandals in finance and the general corporate world, I was intrigued by the idea that the language of a company annual report could carry a forensic signature of cheating or irresponsibility. A case of read what I say, not the numbers I report.
Maximum drawdown is the maximum mark-to-market loss of a portfolio or security over a given period and is a widely used risk management metric. The goal of the project is to see if sharp and extreme equity drawdown, as a proxy for bankruptcy, credit risk and governance, can be predicted from the text of a company’s annual report and to compare this with the performance from more traditional measures using financial metrics (FIN) and market indicators (MKT).
Formal Set-up
The problem is set up as a binary classification problem with the target being the maximum rolling 20-day drawdown of the equity price in the year following the release of the annual report. The target registers an “almost bankrupt” (positive) event when the drawdown is greater than or equal to 80%. The feature set is derived from preprocessing the text of company annual reports and representing these in a term-frequency-inverse-document-frequency (tf-idf) matrix.
Databases
Sharadar offers a reasonably priced subscription covering over 20 years of history across 14,000+ US companies and is accessible through the Quandl API. Importantly for our application, the Sharadar stock database includes bankrupt and other delisted companies. Public companies are required to file annual reports (10K) with the SEC and these are available from the EDGAR database on the SEC website. An existing Python package was used to scrape this data. The stock price database provided 160,926 potential target events of which 38,807 could be matched with the downloaded annual report database.
Text Preprocessing
Given the multi-decade span of the data, the annual reports are in multiple formats including text, html and XBRL. As a result, Regex was used to parse the reports and remove special characters:
def remove_html_tags_char(text):    '''Takes in string and removes special characters '''        #Define special Chars    clean1 = re.compile('\n')                   clean2 = re.compile('\r')                   clean3 = re.compile('&nbsp;')               clean4 = re.compile('&#160;')    clean5 = re.compile('  ')    #Define html tags    clean6 = re.compile('<.*?>')    #remove special characters and html tags    text = re.sub(clean1,' ', text)    text = re.sub(clean2,' ',text)      text = re.sub(clean3,' ',text)     text = re.sub(clean4,' ',text)     text = re.sub(clean5,' ',text)     text = re.sub(clean6,' ',text)     # check spacing    final_text = ' '.join(text.split())return final_text
In keeping with modern methods, the text was minimally processed with no lemmatization, stemming or stop-word removal. This follows the idea that with large enough data sets, one should let the model determine whether these nuances (e.g. tense) are important for the problem at hand.
Data Exploration
The data is highly imbalanced:
Not surprisingly, “almost bankruptcy” spikes over US recessions (relationship to actual US GDP is more nuanced- more on GitHub):
Like many other market price variables, auto-correlation or persistence is evident in equity drawdown:
This last observation leads to the introduction of the market (MKT) model for baseline comparative purposes. MKT simply uses a company’s prior annual drawdown quartile ranking as the sole feature.
Cross-Validation
Method
Data from 2015 onwards is used as the hold-out set while the training data runs from 1997 to 2014. An expanding window method is used with the CV set split into 5 equal parts:
The tf-idf vectorizer is formed on the CV training data and transform is then used to convert the testing documents into its tf-idf matrix. Macro (harmonic) recall is used as the CV score and is weighted in proportion to the amount of data in the training set (i.e. CV1 has weight 10%, CV2 has weight 20% and so on). In cases requiring a tie-break, the standard deviation of the score across CV sets was used.
Structure
Undersampling, oversampling and class-weights were used to tackle data imbalance. Another sample bias considered was the potential for an unequal distribution of financial statements over time to harm the generalization of the model. Indeed, we might expect the changing regulatory, accounting, legal and economic landscape of the last 23 years to be reflected in the language and structure of the annual reports. In order to test for this, validation was performed on (i) a random selection of the majority class data and (ii) a preprocessed sample that kept the ratio of the majority class events per year equal to that of the minority class (time equalization). Finally, various probability threshold values and ensembles were tested.
Key Results
· Undersampling performed better than oversampling
· Time equalization improves the undersampling score from 62% to 66%
· Gradient boosting performs better than Random Forest and Logarithmic Regression
· Undersampling favored positive recall (sensitivity) while oversampling favored negative recall (specificity)
· The optimal model is an ensemble of oversampling / undersampling in a ratio of 25% / 75% with gradient boosting used for both models
Holdout Testing
Results
The optimal model (NLP) was applied to the holdout set on an annual expanding window basis. Aggregate results were then compared to baseline financial ratio (FIN) and market (MKT) models:
The NLP confusion matrix is:
Analysis
One of the benefits of having a continuous variable underlying the binary target is that we can compute the cost of model errors by calculating the drawdown statistics for each class in the confusion matrix:
Despite the relatively high proportion of false positives and the associated low precision, the model errors reveal interesting information about the classification space
This can be more practically fleshed out in the domain space by forming evenly weighted portfolios of predicted true, predicted negative and all:
Interpretation
Interpretation of these models is somewhat challenging, but we can look at what words in the holdout set have the highest probabilities on a standalone basis. This is done by forming a document matrix where each document only has one unique word and then running this through model prediction. The caveat is that a word can have a high probability while being specific to one or very few documents. In this respect, not all words will be generally predictive.
Starting with the metadata, the maximum single word probability is below 50% and 99.9% of the words are between 20 and 30%. This indicates that it is the combination of words found in the document and not single implicating words that produce documents of high probability.
The below list of the top words was manually sorted into categories. Of these categories: accounting/credit, consultant speak/business reorganization and negative sentiment words are intuitively pleasing.
Further Thoughts
·It is somewhat surprising to see a relatively simple model such as tf-idf perform so well in comparison to a baseline financial ratio model. The top word categories of credit and business reorganization suggests the model is good at picking up on similar sort of information to that contained in the financial ratios. The next phase of the project is to explore the language related to almost bankrupt events with strong financial ratios. This focus on “additionality” will likely require more sophisticated algorithms (such as BERT) better able to parse context but might be able to shed light on accounting fraud or other dishonest practices absent from the financials.
· Identifying weak companies is one thing but knowing when that weakness will manifest in a positive event is far harder and will also depend on the complex external environment. This is a reasonable interpretation of the high false positive rate. It would be fascinating to see if the text of the reports contained more information on timing than already extracted. One avenue to try would be cosine similarity but this is likely to suffer from the curse of dimensionality. Another approach would be including the annual change in the tf-idf matrix itself for individual companies in the features set. On the aggregate level, one could try to test whether the aggregate annual tf-idf matrix is predictive of the change in the number of almost bankruptcies in the following year.
Github
· The model, together with other project files, are available on GitHub. Have fun and please feel free to share feedback!