NLP in the Stock Market. Leveraging sentiment analysis on 10-k... | by Roshan Adusumilli | Towards Data Science

Machine learning models implemented in trading are often trained on historical stock prices and other quantitative data to predict future stock prices. However, natural language processing (NLP) enables us to analyze financial documents such as 10-k forms to forecast stock movements. 10-k forms are annual reports filed by companies to provide a comprehensive summary of their financial performance (these reports are mandated by the Securities and Exchange Commission). Combing through these reports is often tedious for investors. Through sentiment analysis, a subfield of natural language processing, investors can quickly understand if the tone of the report is positive, negative, or litigious etc. The overall sentiment expressed in the 10-k form can then be used to help investors decide if they should invest in the company.
Natural language processing is a branch of artificial intelligence concerned with teaching computers to read and derive meaning from language. Since language is so complex, computers have to be taken through a series of steps before they can comprehend text. The following is a quick explanation of the steps that appear in a typical NLP pipeline.
Sentence Segmentation The text document is segmented into individual sentences.TokenizationOnce the document is broken into sentences, we further split the sentences into individual words. Each word is called a token, hence the name tokenization.Parts-of-Speech-TaggingWe input each token as well as a few words around it into a pre trained part-of-speech classification model to receive the part-of-speech for the token as an output.LemmatizationWords often appear in different forms while referring to the same object/action. To prevent the computer from thinking of different forms of a word as different words, we perform lemmatization, the process of grouping together various inflections of a word to analyze them as a single item, identified by the word’s lemma (how the word appears in the dictionary).Stop WordsExtremely common words such as “and”, “the” and “a” don’t provide any value, so we identify them as stop words to exclude them from any analysis performed on the text.Dependency ParsingAssign a syntactic structure to sentences and make sense of how the words in the sentence relate to each other by feeding the words to a dependency parser.Noun PhrasesGrouping the noun phrases in a sentence together can help simplify sentences for cases when we don’t care about adjectives.Named Entity RecognitionA Named Entity Recognition model can tag objects such as people’s names,company names, and geographic locations.Coreference ResolutionSince NLP models analyze individual sentences, they become confused by pronouns referring to nouns from other sentences. To solve this problem, we employ coreference resolution which tracks pronouns across sentences to avoid confusion.
Sentence Segmentation The text document is segmented into individual sentences.
TokenizationOnce the document is broken into sentences, we further split the sentences into individual words. Each word is called a token, hence the name tokenization.
Parts-of-Speech-TaggingWe input each token as well as a few words around it into a pre trained part-of-speech classification model to receive the part-of-speech for the token as an output.
LemmatizationWords often appear in different forms while referring to the same object/action. To prevent the computer from thinking of different forms of a word as different words, we perform lemmatization, the process of grouping together various inflections of a word to analyze them as a single item, identified by the word’s lemma (how the word appears in the dictionary).
Stop WordsExtremely common words such as “and”, “the” and “a” don’t provide any value, so we identify them as stop words to exclude them from any analysis performed on the text.
Dependency ParsingAssign a syntactic structure to sentences and make sense of how the words in the sentence relate to each other by feeding the words to a dependency parser.
Noun PhrasesGrouping the noun phrases in a sentence together can help simplify sentences for cases when we don’t care about adjectives.
Named Entity RecognitionA Named Entity Recognition model can tag objects such as people’s names,company names, and geographic locations.
Coreference ResolutionSince NLP models analyze individual sentences, they become confused by pronouns referring to nouns from other sentences. To solve this problem, we employ coreference resolution which tracks pronouns across sentences to avoid confusion.
For a more in-depth description of NLP: read this
After going through these steps, our text is ready for analysis. Now that we understand NLP better, let’s take a look at my code for the project (project 5 from Udacity’s AI for Trading course). Click here to view the full Github repository
First, we make the necessary imports; project_helper contains various utility and graph functions.
import nltkimport numpy as npimport pandas as pdimport pickleimport pprintimport project_helperfrom tqdm import tqdm
Then we download the stopwords corpus for removing stopwords and wordnet corpus for lemmatizing.
nltk.download('stopwords')nltk.download('wordnet')
10-k documents include information such as company history, organizational structure, executive compensation, equity, subsidiaries, and audited financial statements. To lookup 10-k documents, we use each company’s unique CIK (Central Index Key).
cik_lookup = {    'AMZN': '0001018724',    'BMY': '0000014272',       'CNP': '0001130310',    'CVX': '0000093410',    'FL': '0000850209',    'FRT': '0000034903',    'HON': '0000773840'}
Now we pull a list of filed 10-ks from the SEC and display Amazon data as an example.
sec_api = project_helper.SecAPI()from bs4 import BeautifulSoupdef get_sec_data(cik, doc_type, start=0, count=60):    rss_url = 'https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany' \        '&CIK={}&type={}&start={}&count={}&owner=exclude&output=atom' \        .format(cik, doc_type, start, count)    sec_data = sec_api.get(rss_url)    feed = BeautifulSoup(sec_data.encode('ascii'), 'xml').feed    entries = [        (            entry.content.find('filing-href').getText(),            entry.content.find('filing-type').getText(),            entry.content.find('filing-date').getText())        for entry in feed.find_all('entry', recursive=False)]return entriesexample_ticker = 'AMZN'sec_data = {}for ticker, cik in cik_lookup.items():    sec_data[ticker] = get_sec_data(cik, '10-K')pprint.pprint(sec_data[example_ticker][:5])
We received a list of urls pointing to files containing metadata related to each filling. Metadata isn’t relevant to us so we pull the filling by replacing the url with the filling url. Let’s view download progress by using tqdm and look at an example document.
raw_fillings_by_ticker = {}for ticker, data in sec_data.items():    raw_fillings_by_ticker[ticker] = {}    for index_url, file_type, file_date in tqdm(data, desc='Downloading {} Fillings'.format(ticker), unit='filling'):        if (file_type == '10-K'):            file_url = index_url.replace('-index.htm', '.txt').replace('.txtl', '.txt')                                    raw_fillings_by_ticker[ticker][file_date] = sec_api.get(file_url)print('Example Document:\n\n{}...'.format(next(iter(raw_fillings_by_ticker[example_ticker].values()))[:1000]))
Break downloaded files into their associated documents, which are sectioned off in the fillings with the tags <DOCUMENT> for the start of each document and </DOCUMENT> for the end of each document.
import redef get_documents(text):    extracted_docs = []        doc_start_pattern = re.compile(r'<DOCUMENT>')    doc_end_pattern = re.compile(r'</DOCUMENT>')           doc_start_is = [x.end() for x in      doc_start_pattern.finditer(text)]    doc_end_is = [x.start() for x in doc_end_pattern.finditer(text)]        for doc_start_i, doc_end_i in zip(doc_start_is, doc_end_is):            extracted_docs.append(text[doc_start_i:doc_end_i])        return extracted_docsfilling_documents_by_ticker = {}for ticker, raw_fillings in raw_fillings_by_ticker.items():    filling_documents_by_ticker[ticker] = {}    for file_date, filling in tqdm(raw_fillings.items(), desc='Getting Documents from {} Fillings'.format(ticker), unit='filling'):        filling_documents_by_ticker[ticker][file_date] = get_documents(filling)print('\n\n'.join([    'Document {} Filed on {}:\n{}...'.format(doc_i, file_date, doc[:200])    for file_date, docs in filling_documents_by_ticker[example_ticker].items()    for doc_i, doc in enumerate(docs)][:3]))
Define the get_document_type function to return the type of document given.
def get_document_type(doc):        type_pattern = re.compile(r'<TYPE>[^\n]+')        doc_type = type_pattern.findall(doc)[0][len('<TYPE>'):]         return doc_type.lower()
Filter out the non 10-k documents from the fillings using the get_document_type function.
ten_ks_by_ticker = {}for ticker, filling_documents in filling_documents_by_ticker.items():    ten_ks_by_ticker[ticker] = []    for file_date, documents in filling_documents.items():        for document in documents:            if get_document_type(document) == '10-k':                ten_ks_by_ticker[ticker].append({                    'cik': cik_lookup[ticker],                    'file': document,                    'file_date': file_date})project_helper.print_ten_k_data(ten_ks_by_ticker[example_ticker][:5], ['cik', 'file', 'file_date'])
Remove the html and make all text lowercase to clean up the document text.
def remove_html_tags(text):    text = BeautifulSoup(text, 'html.parser').get_text()        return textdef clean_text(text):    text = text.lower()    text = remove_html_tags(text)        return text
Clean up the documents using the clean_text function.
for ticker, ten_ks in ten_ks_by_ticker.items():    for ten_k in tqdm(ten_ks, desc='Cleaning {} 10-Ks'.format(ticker), unit='10-K'):        ten_k['file_clean'] = clean_text(ten_k['file'])project_helper.print_ten_k_data(ten_ks_by_ticker[example_ticker][:5], ['file_clean'])
Now we lemmatize all the data.
from nltk.stem import WordNetLemmatizerfrom nltk.corpus import wordnetdef lemmatize_words(words):    lemmatized_words = [WordNetLemmatizer().lemmatize(word, 'v') for word in words]        return lemmatized_wordsword_pattern = re.compile('\w+')for ticker, ten_ks in ten_ks_by_ticker.items():    for ten_k in tqdm(ten_ks, desc='Lemmatize {} 10-Ks'.format(ticker), unit='10-K'):        ten_k['file_lemma'] = lemmatize_words(word_pattern.findall(ten_k['file_clean']))project_helper.print_ten_k_data(ten_ks_by_ticker[example_ticker][:5], ['file_lemma'])
Remove the stopwords.
from nltk.corpus import stopwordslemma_english_stopwords = lemmatize_words(stopwords.words('english'))for ticker, ten_ks in ten_ks_by_ticker.items():    for ten_k in tqdm(ten_ks, desc='Remove Stop Words for {} 10-Ks'.format(ticker), unit='10-K'):        ten_k['file_lemma'] = [word for word in ten_k['file_lemma'] if word not in lemma_english_stopwords]print('Stop Words Removed')
Use the Loughran-McDonald sentiment word lists to perform sentiment analysis on the 10-ks (this was specifically built for textual analysis related finance).
sentiments = ['negative', 'positive', 'uncertainty', 'litigious', 'constraining', 'interesting']sentiment_df = pd.read_csv('loughran_mcdonald_master_dic_2018.csv')sentiment_df.columns = [column.lower() for column in sentiment_df.columns] # Lowercase the columns for ease of use# Remove unused informationsentiment_df = sentiment_df[sentiments + ['word']]sentiment_df[sentiments] = sentiment_df[sentiments].astype(bool)sentiment_df = sentiment_df[(sentiment_df[sentiments]).any(1)]# Apply the same preprocessing to these words as the 10-k wordssentiment_df['word'] = lemmatize_words(sentiment_df['word'].str.lower())sentiment_df = sentiment_df.drop_duplicates('word')sentiment_df.head()
Generate sentiment bag of words from the 10-k documents using the sentiment word lists. The bag of words counts the number of sentiment words in each doc.
from collections import defaultdict, Counterfrom sklearn.feature_extraction.text import CountVectorizerdef get_bag_of_words(sentiment_words, docs):    vec = CountVectorizer(vocabulary=sentiment_words)    vectors = vec.fit_transform(docs)    words_list = vec.get_feature_names()    bag_of_words = np.zeros([len(docs), len(words_list)])        for i in range(len(docs)):        bag_of_words[i] = vectors[i].toarray()[0]return bag_of_words.astype(int)sentiment_bow_ten_ks = {}for ticker, ten_ks in ten_ks_by_ticker.items():    lemma_docs = [' '.join(ten_k['file_lemma']) for ten_k in ten_ks]        sentiment_bow_ten_ks[ticker] = {        sentiment: get_bag_of_words(sentiment_df[sentiment_df[sentiment]]['word'], lemma_docs)        for sentiment in sentiments}project_helper.print_ten_k_data([sentiment_bow_ten_ks[example_ticker]], sentiments)
Now that we have the bag of words, we can convert it into a boolean array and calculate the jaccard similarity. The jaccard similarity is defined as the size of intersection divided by the size of union of two sets. For example, the jaccard similarity between two sentences is the number of common words between the two sentences divided by the total number of unique words in both sentences combined. The closer the jaccard similarity value is to 1, the more similar the sets are. To more easily understand our calculations, we plot the jaccard similarities.
from sklearn.metrics import jaccard_similarity_scoredef get_jaccard_similarity(bag_of_words_matrix):        jaccard_similarities = []    bag_of_words_matrix = np.array(bag_of_words_matrix, dtype=bool)        for i in range(len(bag_of_words_matrix)-1):            u = bag_of_words_matrix[i]            v = bag_of_words_matrix[i+1]                  jaccard_similarities.append(jaccard_similarity_score(u,v))            return jaccard_similarities# Get dates for the universefile_dates = {    ticker: [ten_k['file_date'] for ten_k in ten_ks]    for ticker, ten_ks in ten_ks_by_ticker.items()}jaccard_similarities = {    ticker: {        sentiment_name: get_jaccard_similarity(sentiment_values)        for sentiment_name, sentiment_values in ten_k_sentiments.items()}    for ticker, ten_k_sentiments in sentiment_bow_ten_ks.items()}project_helper.plot_similarities(    [jaccard_similarities[example_ticker][sentiment] for sentiment in sentiments],    file_dates[example_ticker][1:],    'Jaccard Similarities for {} Sentiment'.format(example_ticker),    sentiments)
From the sentiment word lists, let’s generate sentiment term frequency–inverse document frequency (TFIDF) from the 10-k documents. TFIDF is an information retrieval technique used to reveal how often a word/term appears in the chosen collection of text. Each term is assigned a term frequency (TF) and inverse document frequency (IDF) score. The product of these scores is referred to as the TFIDF weight of the term. Higher TFIDF weights indicate rarer terms and lower TFIDF scores indicate more common terms.
from sklearn.feature_extraction.text import TfidfVectorizerdef get_tfidf(sentiment_words, docs):        vec = TfidfVectorizer(vocabulary=sentiment_words)    tfidf = vec.fit_transform(docs)        return tfidf.toarray()sentiment_tfidf_ten_ks = {}for ticker, ten_ks in ten_ks_by_ticker.items():    lemma_docs = [' '.join(ten_k['file_lemma']) for ten_k in ten_ks]        sentiment_tfidf_ten_ks[ticker] = {        sentiment: get_tfidf(sentiment_df[sentiment_df[sentiment]]['word'], lemma_docs)        for sentiment in sentiments}project_helper.print_ten_k_data([sentiment_tfidf_ten_ks[example_ticker]], sentiments)
From our TFIDF values, we can calculate the cosine similarity and plot it over time. Similar to jaccard similarity, cosine similarity is a metric used to determine how similar documents are. Cosine similarity calculates similarity irrespective of size by measuring the cosine of the angle between two vectors projected in a multi-dimensional space. For textual analysis, the two vectors used are usually arrays containing the word counts of two documents.
from sklearn.metrics.pairwise import cosine_similaritydef get_cosine_similarity(tfidf_matrix):        cosine_similarities = []            for i in range(len(tfidf_matrix)-1):        cosine_similarities.append(cosine_similarity(tfidf_matrix[i].reshape(1, -1),tfidf_matrix[i+1].reshape(1, -1))[0,0])        return cosine_similaritiescosine_similarities = {    ticker: {        sentiment_name: get_cosine_similarity(sentiment_values)        for sentiment_name, sentiment_values in ten_k_sentiments.items()}    for ticker, ten_k_sentiments in sentiment_tfidf_ten_ks.items()}project_helper.plot_similarities(    [cosine_similarities[example_ticker][sentiment] for sentiment in sentiments],    file_dates[example_ticker][1:],    'Cosine Similarities for {} Sentiment'.format(example_ticker),    sentiments)
Now we’ll evaluate the alpha factors by comparing them against the yearly pricing for stocks. We can download the pricing data from QuoteMedia.
pricing = pd.read_csv('yr-quotemedia.csv', parse_dates=['date'])pricing = pricing.pivot(index='date', columns='ticker', values='adj_close')pricing
Alphalens, a python library for performance analysis of alpha factors, uses dataframes, so we have to convert our dictionary into a dataframe.
cosine_similarities_df_dict = {'date': [], 'ticker': [], 'sentiment': [], 'value': []}for ticker, ten_k_sentiments in cosine_similarities.items():    for sentiment_name, sentiment_values in ten_k_sentiments.items():        for sentiment_values, sentiment_value in enumerate(sentiment_values):            cosine_similarities_df_dict['ticker'].append(ticker)            cosine_similarities_df_dict['sentiment'].append(sentiment_name)            cosine_similarities_df_dict['value'].append(sentiment_value)            cosine_similarities_df_dict['date'].append(file_dates[ticker][1:][sentiment_values])cosine_similarities_df = pd.DataFrame(cosine_similarities_df_dict)cosine_similarities_df['date'] = pd.DatetimeIndex(cosine_similarities_df['date']).yearcosine_similarities_df['date'] = pd.to_datetime(cosine_similarities_df['date'], format='%Y')cosine_similarities_df.head()
Before we can utilize many of the alphalens functions, we need to align the indices and convert the time to unix timestamp.
import alphalens as alfactor_data = {}skipped_sentiments = []for sentiment in sentiments:    cs_df = cosine_similarities_df[(cosine_similarities_df['sentiment'] == sentiment)]    cs_df = cs_df.pivot(index='date', columns='ticker', values='value')        try:        data = al.utils.get_clean_factor_and_forward_returns(cs_df.stack(), pricing.loc[cs_df.index], quantiles=5, bins=None, periods=[1])        factor_data[sentiment] = data    except:        skipped_sentiments.append(sentiment)if skipped_sentiments:    print('\nSkipped the following sentiments:\n{}'.format('\n'.join(skipped_sentiments)))factor_data[sentiments[0]].head()
We also have to create factor dataframes with unix time for compatibility with alphalen’s factor_rank_autocorrelation and mean_return_by_quantile functions.
unixt_factor_data = {    factor: data.set_index(pd.MultiIndex.from_tuples(        [(x.timestamp(), y) for x, y in data.index.values],        names=['date', 'asset']))    for factor, data in factor_data.items()}
Let’s take a look at the factor returns over time
ls_factor_returns = pd.DataFrame()for factor_name, data in factor_data.items():    ls_factor_returns[factor_name] = al.performance.factor_returns(data).iloc[:, 0](1 + ls_factor_returns).cumprod().plot()
As expected, 10-k reports expressing positive sentiment produced the most gains while 10-k reports containing negative sentiment resulted in the most losses.
Using the Factor Rank Autocorrelation we can analyze how stable the alphas are over time. We want the alpha ranks to remain relatively same from period to period.
ls_FRA = pd.DataFrame()for factor, data in unixt_factor_data.items():    ls_FRA[factor] = al.performance.factor_rank_autocorrelation(data)ls_FRA.plot(title="Factor Rank Autocorrelation")
Finally, let’s calculate the Sharpe ratio, which is the average return minus the risk-free return divided by the standard deviation of return on an investment.
daily_annualization_factor = np.sqrt(252)  (daily_annualization_factor * ls_factor_returns.mean() / ls_factor_returns.std()).round(2)
A Sharpe ratio of 1 is considered acceptable, a ratio of 2 is very good, and a ratio of 3 is excellent. As expected, we can see that positive sentiment correlates to a high Sharpe ratio and negative sentiment correlates to a low Sharpe ratio. The other sentiments also correlated to high Sharpe ratios. However, since so many complex factors influence the price of stocks it is infinitely harder to replicate these returns in the real world.
[1] Udacity, Artificial Intelligence for Trading, Github
Don’t leave yet!
I’m Roshan, a 16 year old passionate about the intersection of artificial intelligence and finance. For a broad view of AI in finance, check out this article: https://towardsdatascience.com/artificial-intelligence-and-its-application-in-finance-9f1e0588e777.
Reach out to me on Linkedin: https://www.linkedin.com/in/roshan-adusumilli-96b104194/