Visualizing the Stock Market with Tableau | by Tony Yiu | Towards Data Science

Tableau is a critical data visualization tool that belongs in every data analyst’s and data scientist’s toolkit. Today we play around with stock market data in order to explore how Tableau can help us dissect and better understand our data.
We can grab stock market data from Quandl. We’re going to focus on the S&P 500, a market cap weighted index of large cap American stocks (basically, the biggest American companies).
Here is the Python code for grabbing the data (note that you will need your own Quandl API key and that the Sharadar data requires a paid subscription). I’ve also added comments in the code below, but the following snippet uses Quandl’s API to grab financial data for all publicly listed companies in Sharadar’s database (financial data is stuff like revenues, net income, debt, return on invested capital, and so on). This financial data is stored in the dataframe stock_df.
import quandlquandl.ApiConfig.api_key = 'your_qandl_api_key'import numpy as npimport pandas as pd# Download financial data from Quandl/Sharadartable = quandl.get_table('SHARADAR/SF1', paginate=True)# Grab the most recent annual data ('MRY' denotes annual data)stock_df = table[(table['calendardate'] == '2018-12-31 00:00:00') & (table['dimension']=='MRY')]
However, we don’t need data on every company — we just want the S&P 500. To get the S&P 500 tickers, we turn to web scraping and Wikipedia. The following lines of code web scrape a Wikipedia table that lists the tickers and industries of every company that is currently in the S&P 500 and stores it in the variable called page_content.
# Scrape S&P 500 tickers from Wikipediafrom bs4 import BeautifulSoupimport requestsimport repage_link = 'https://en.wikipedia.org/wiki/List_of_S%26P_500_companies'page_response = requests.get(page_link, timeout=1000)page_content = BeautifulSoup(page_response.content, 'lxml')
Next we use some parsing code to grab ticker and industry from page_content, which is a BeautifulSoup data structure. The code can be summarized as doing the following:
Loop through all the ‘tr’ (I believe it denotes a table row in HTML) tagged items in page_content.
Grab the ticker as a string (the .text converts the item to string format) and store it in the list tickers.
Grab the industry as a string and store it in the list industries.
# From web scraped Wikipedia content on the S&P 500, grab the ticker and industry for each firmtickers = []industries = []for i, val in enumerate(page_content.find_all('tr')):    if i > 0:        try:            # Here is where I grab the ticker            tickers.append(val.find_all('a')[0].text)            # Here is where I grab the industry            industries.append(val.find_all('td')[3].text.replace('\n', ''))        except:            pass
Finally, we need to do a bit of data manipulation to join up our ticker and industry data (from Wikipedia) with our financial data (from Quandl). The first two blocks of code (as detailed in the comments) store the tickers that we want into a dataframe called sp_df.
The last block of code uses the Pandas merge method to join sp_df, which contains our ticker and industry data, with stock_df, which contains our financial data. We store the resulting dataframe, merged_df, in a .csv file that we can now load into Tableau (I use Tableau Public, which is free). Hurray done!
# After ticker ZTS, the rest of the table entries are for acquisitions/mergerslast_pos = tickers.index('ZTS') + 1sp_tickers = tickers[:last_pos]sp_industries = industries[:last_pos]# Create a new dataframe for S&P 500 and merge in Quandl datasp_df = pd.DataFrame()sp_df['tickers'] = sp_tickerssp_df['industries'] = sp_industriesmerged_df = sp_df.merge(stock_df, left_on='tickers', right_on='ticker', how='left')merged_df.to_csv('sp500.csv', index=False)
Data visualization sounds fun and easy but it’s not. A lot of thought goes into all those cool dashboards and interactive charts that we see. While I am no expert, before I start slapping numbers onto graphs, I first try to answer the following questions:
What insight are we trying to uncover through our visualization? Pretty charts are nice and all but if there is no insight to draw from them then they are nothing more than empty shapes and colors.Is the visual meant to be exploratory or answer a specific question? These two objectives might seem similar but they are not. An exploratory visualization can afford to be more general and contain lots of information — as the objective is to provoke thought and discussion. It is meant to be stared at, digested gradually, and different people can arrive at different conclusions upon studying one. On the other hand, visualizations meant to answer a specific question need to do exactly that — they are meant to be unambiguous supporters of whatever point you are trying to make.
What insight are we trying to uncover through our visualization? Pretty charts are nice and all but if there is no insight to draw from them then they are nothing more than empty shapes and colors.
Is the visual meant to be exploratory or answer a specific question? These two objectives might seem similar but they are not. An exploratory visualization can afford to be more general and contain lots of information — as the objective is to provoke thought and discussion. It is meant to be stared at, digested gradually, and different people can arrive at different conclusions upon studying one. On the other hand, visualizations meant to answer a specific question need to do exactly that — they are meant to be unambiguous supporters of whatever point you are trying to make.
Let’s try to answer these questions now:
We want to use our stock data to look for attractive industries or individual companies that might be worth investing in.We will focus on exploratory visuals in this post. Picking stocks is a complicated and subjective endeavor. Our aim here is not to tell people what stocks to buy; but rather we hope our visualizations can help them identify and drill down into more specific areas of interest.
We want to use our stock data to look for attractive industries or individual companies that might be worth investing in.
We will focus on exploratory visuals in this post. Picking stocks is a complicated and subjective endeavor. Our aim here is not to tell people what stocks to buy; but rather we hope our visualizations can help them identify and drill down into more specific areas of interest.
Let’s start off by looking at a Tableau treemap of the S&P 500. The S&P 500 is a cap weighted index — that means each company’s representation in the index is proportional to its market cap (market cap is the sum value of all the company’s shares, in other words what it would cost to buy the entire firm). In the following treemap, the size and color of each rectangle corresponds to that company’s representation in the S&P 500.
There are some familiar faces at the top — Apple, Microsoft, Amazon, Google, Berkshire Hathaway make up the top 5. Interestingly, these top 5 companies make up 17% of the S&P 500’s market cap and the top 15 companies make up 30% of the total market cap. Note that market cap alone is not enough to explain what drives the S&P 500’s price changes — the volatility of a company’s share price matters too. For example, even though Facebook (FB) has a lower market cap than Microsoft (MSFT), Facebook’s share price has been more volatile in the past 12 months as it has attempted to navigate a slew of scandals. Thus, over the last year, Facebook’s stock has most likely been a greater driver of the S&P 500 than Microsoft’s has.
Owning a share of a company is equivalent to owning a portion (however small) of that same company. For example, if we were fabulously rich and bought shares in Google equivalent to 1% of the firm, then as minority owners, we would be entitled to 1% of the profits. Of course, as minority owners, we are also forced to entrust the management and investment of these profits to Google’s upper management and board (which is why we don’t get a profit sharing check from Google every quarter even if we own the shares).
So if a firm’s profits (a.k.a. net income) are what we ultimately hope to gain when we buy its shares, perhaps we should redo our treemap in terms of profits. In the following treemap, the size and color of each rectangle corresponds to the magnitude of that company’s profits (a.k.a. net income).
While Apple and Google are still in the top 5, we now have some new faces — J.P. Morgan (JPM), Bank of America (BAC), and Wells Fargo (WFC). Despite all the hype over how tech is upending traditional industries, the banking industry (as traditional an industry as there is) continues to earn massive profits. Intel (INTC) the forgotten tech titan, Exxon Mobil (XOM) the oil giant, and AT&T (T) the telecommunications conglomerate also still earn tons of money despite being somewhat forgotten in today’s innovation obsessed market.
Personally I believe the best long term investment strategy is to “buy low and sell high” (easier said than done). But how do we do that? At its most basic level, for each dollar that we pay, we should be trying to purchase as many dollars in profits as possible. The previous visualization showed us that unloved industries may sometimes harbor under-appreciated sources of profit. Let’s shift to looking at industries instead of companies to see whether we can find some good profit ponds to fish in.
The treemap below breaks down each industry in terms of median market cap (the area of each rectangle) and median profits (the color of each rectangle). We use medians because we want to know that if I were to pick a company at random from a particular industry bucket, how much is it likely to cost (market cap) and how profitable is it likely to be (net income).
Before we discuss results, we should first understand the proper way to think about this treemap. We are using it to eyeball the relationship between median market cap (the price we pay) and median net income (the profits we get). If the price of a sector were perfectly correlated with the profits of the firms inside it, then we would expect the biggest rectangles to be the most green and for the color to transition to red as the rectangles decrease in size. We obviously don’t see that here, so it looks like there are potential opportunities.
The treemap highlights Financials (banks, insurance companies, etc.) and Communication Services (Google, Facebook, AT&T, Disney, etc.) as potentially attractive sectors in which to search for cheap stocks — because they are middle of the pack in terms of rectangle size (median market cap) but very green (high median profits).
Meanwhile, Real Estate, in deep red, seems like a definite no go. Healthcare also seems like a sector to avoid as it owns both the highest median market cap along with near “bottom of the pack” median profits (bad bang for our buck).
Of course, these are just quick rules of thumb for identifying potential investments — much more due diligence should be done before an actual investment is made.
Surprisingly the Information Technology sector, which contains heavy hitters like Apple and Microsoft, does not come off looking great in our previous analysis — it is expensive in terms of price but “back of the pack” in terms of median profits. What gives?
There are two factors at play:
In late 2018, Google and Facebook were moved from Information Technology to Communication Services. Those are two massively profitable companies that people associate deeply with tech. If they were still in the Information Technology sector, then both its market cap and profits would be significantly higher.The level of profits is not the only thing investors care about. How those profits are earned is also extremely important — things like profit margin (what proportion of sales ultimately translate into profits) and return on investment (how much money we need to put into a business in order to generate a given level of profits) matter a lot too.
In late 2018, Google and Facebook were moved from Information Technology to Communication Services. Those are two massively profitable companies that people associate deeply with tech. If they were still in the Information Technology sector, then both its market cap and profits would be significantly higher.
The level of profits is not the only thing investors care about. How those profits are earned is also extremely important — things like profit margin (what proportion of sales ultimately translate into profits) and return on investment (how much money we need to put into a business in order to generate a given level of profits) matter a lot too.
Let’s use a Tableau scatter plot to check out the relationship between P/E ratios and ROIC:
P/E ratio is a popular measure of how expensive a company’s stock is. It is simply the company’s market capitalization divided by its net income — in other words, how much does it cost us to buy $1 of a particular company’s earnings. The higher the P/E ratio, all other things equal, the more expensive a stock is perceived to be.
ROIC stands for Return on Capital Invested and represents the amount of profit we can generate from a $1 investment into the business. Invested capital is a measure of the amount of capital it takes to run a certain business. For example, a capital intensive business would be something like Toyota, which requires massive factories to manufacture its vehicles. On the opposite side of the spectrum, software is a capital light business — once it is developed, it costs very little in terms of fixed costs or infrastructure (relative to the profits generated) to maintain and deploy something like Microsoft’s Office (Word, Excel, PowerPoint, Outlook).
Check out our scatter plot to the left. It shows that there is a positive correlation between a stock’s P/E ratio (how expensive it is) and its ROIC. This makes sense — investors like high ROIC companies because they generally require little upfront investment to generate significant profits (companies that require large upfront investments are viewed as risky because we might never get paid back).
And the Information Technology sector has by far the highest median ROIC. This also makes sense as this sector is filled with software firms, which as we previously observed are very capital light. So it’s not high current profits that attract investors to Information Technology (and drive its high P/E ratio and market capitalizations) but rather the capital light nature of the business and the hope that relatively small investments today will yield high growth and profits tomorrow (everyone loves a business model that is easy and cheap to scale).
Will that happen? I have no idea but the market is currently making a big bet that it will — so if you are looking for good odds, you may want to look elsewhere.
Finally, note that the Financials sector has the lowest ROIC. This plus the perceived riskiness of banks (2008 anyone?) probably drives their low valuation. But banks use capital differently — instead of borrowing money to build a factory, banks borrow money from folks like you and me in order to lend it back out at higher rates.
Usually high invested capital implies that high fixed, upfront costs are needed to operate the business. This is not the case for banks — despite having extremely high levels of invested capital (due to high debt), banks have very little in the ways of fixed costs. This high debt does make them riskier and perhaps deserving of a P/E ratio discount, but we should not rule banks out as investments just because of their low ROICs — instead, we should seek to understand what drives these low ROICs and whether it is a real concern or not.
Nice, we made it to the end! Honestly, there is still so much more we can explore. The stock market and corporate strategy are two incredibly deep and rich subjects; and the question of “what stocks to buy” is an immensely tricky one. We have only begun to scratch the surface in this post.
However, I hope that I’ve demonstrated how visualizations can help us digest complex datasets and begin to tackle challenging problems.
Cheers!
More by me on Finance and Investing:
The Dangers of Shorting Volatility
A Data Science Project I Did on Investing in Lending Club Loans
On Stock Market Downturns
More by me on Data Science:
The Random Forest Classifier
How Neural Networks Work
Principal Components Analysis