Scraping and Exploring The SP500 With R Part 2 | by Bryant Crocker | Towards Data Science

In the first part of this blog post, I walked through scraping a list of SP500 symbols from wikipedia and using that information to pull OHLC (Open, High Low, Close) data for all SP500 tickers from the yahoo finance API.
In the second part of this blog post, I will walk through answering my original question:
Which SP500 assets had the highest average return over the last 3 months?
In part 1, we created tickers_df, a dataframe with all 505 SP500 tickers.
lets take a quick look again at tickers_df:
This data is tidy meaning:
each variable is a column
each observation (or case) is a row
This structure makes the data easy to filter and aggregate, and makes it easier to scale. We can easily extract and visualize one of the assets raw prices over time with just a few lines of code:
ticker = "GOOGL"tickers_df %>%   filter(symbol == !!ticker) %>%   ggplot(aes(date, adjusted))+  geom_line()
We can look at raw price data all day, but it is hard to compare assets that have drastically different prices. Typically, when we analyze assets, we look at the percentage change in prices or returns. Using the tidyquant package, it is extremely easy to calculate returns. In the following snippet, I convert each assets raw adjusted closing prices to returns. This is done using the tq_transmute() function.
daily_sector = tickers_df %>% group_by(security, gics_sector, symbol) %>% tq_transmute(select     = adjusted,               mutate_fun = periodReturn,               period     = "daily") %>%               ungroup()
A few things to note:
The data is still tidy
We have information from both the Yahoo finance API and the wikipedia page that we scraped
With this format it is very easy to aggregate by symbol, security or gics_sector
Don’t forget to ungroup! I did initially
Top Performers:
Let’s go ahead and compute the average daily return by security and then sort by this, to answer my question. I will also compute the standard deviation of the returns. In quantitative finance, the standard deviation of returns is generally referred to as volatility.
avg_return =daily_sector %>%   group_by(security, gics_sector) %>%  summarise(avg_return = round(mean(daily.returns), 4),Volatility =   sd(daily.returns)) %>%         arrange(desc(avg_return), desc(Volatility))avg_return %>% head()
From this table we can quickly see that Dow Inc, the famous American chemical company, has had the highest average daily return over the last 3 months. This table answers my original question.
Let’s visualize this table as a bar graph using ggplot:
avg_return %>% head(20) %>% ggplot(aes(reorder(security, -avg_return), avg_return, fill = avg_return))+  geom_col()+  coord_flip()+  labs(title = "Securities With Highest Average Returns In SP500 Over Past 3 Month", x = "Security", y = "Average Return")+  theme_classic()+  theme(legend.position="none")
Wow Dow Inc.’s average return is much higher than the rest of the market. There may be something big I’m missing here. A quick google search tells me there’s an obvious reason for this. Dow INC has only been traded as an independent chemical company since the end of March, when it formally split from Dowdupont. The average return is likely only this high because it is a newly traded asset and we don’t have many observations.
Looking at the other top performers I notice a few things:
Many of the top performing companies over the past 30 days have been technology companies.
I’m a bit suprised by the strong performance of Chipotle. They have clearly rebounded from their previous controversies around food poisoning
Nvidia and Advanced Micro Devices, two computer hardware manufacturers, have seen higher returns than there competitor Intel, who historically has performed better.
Typically when you answer a question with data, more questions arise. I already know what stocks had the highest return on average over the last few months, but what about volatility? Volatility is one of the most important investment metrics for a security or portfolio.
According to Wikipedia Investors care about volatility for at least eight reasons:
The wider the swings in an investment’s price, the harder emotionally it is to not worry;Price volatility of a trading instrument can define position sizing in a portfolio;When certain cash flows from selling a security are needed at a specific future date, higher volatility means a greater chance of a shortfall;Higher volatility of returns while saving for retirement results in a wider distribution of possible final portfolio values;Higher volatility of return when retired gives withdrawals a larger permanent impact on the portfolio’s value;Price volatility presents opportunities to buy assets cheaply and sell when overpriced;Portfolio volatility has a negative impact on the compound annual growth rate (CAGR) of that portfolioVolatility affects pricing of options, being a parameter of the Black–Scholes model.
The wider the swings in an investment’s price, the harder emotionally it is to not worry;
Price volatility of a trading instrument can define position sizing in a portfolio;
When certain cash flows from selling a security are needed at a specific future date, higher volatility means a greater chance of a shortfall;
Higher volatility of returns while saving for retirement results in a wider distribution of possible final portfolio values;
Higher volatility of return when retired gives withdrawals a larger permanent impact on the portfolio’s value;
Price volatility presents opportunities to buy assets cheaply and sell when overpriced;
Portfolio volatility has a negative impact on the compound annual growth rate (CAGR) of that portfolio
Volatility affects pricing of options, being a parameter of the Black–Scholes model.
Typically, younger investors can take on more volatility, while investors closer to retirement want to be careful about volatility.
Using ggplot2 we can easily plot the relationship between average return and volatility of returns. I will create a scatter plot, using the asset tickers rather than points in the plot.
plot = avg_return %>% ggplot(aes(avg_return, Volatility))+  geom_text(aes(label = symbol), size = 3)+  labs(title = "Average Return vs Volatility Over Last 3 Months In SP500", x = "Average Return", subtitle = "Data Source: Yahoo Finance")+  theme_minimal()    plot
This plot gives a much better idea of what’s happening in the market than our previous bar graph.
A few tickers really stand out compared to the rest of the market
DOW INC has the highest average return and seems to be around market volatility.
AMD and Coty stand out as having above market volatility and return. These might be solid investment opportunities for younger investors
Matel and EA games stand out as having around market returns, but higher volatility. This high volatility gives investors the opportunity to buy when the securities are underpriced and sell when they are overpriced.
The biggest thing that stands out in the top left corner of the figure is the high volatility and low average return of both Fox corporations asset classes
Lets highlight Fox Corporation’s tickers on the graph:
avg_return = avg_return %>%   mutate(Indicator = case_when(symbol %in% c('FOX', 'FOXA') ~ "Fox Corporation",                               TRUE ~ "The Rest of the SP500"))plot = avg_return %>% ggplot(aes(avg_return, Volatility, color = Indicator))+  geom_text(aes(label = symbol), size = 3)+  labs(title = "Average Return vs Volatility Over Last 3 Months In SP500", x = "Average Return", subtitle = "Data Source: Yahoo Finance")+  theme_minimal()    plot
Wow! look at that! I walked into this analysis interested in which companies were performing the best. But, at the end, what really stood out is the Fox Corporation which, appears to be performing quite weak relative to the rest of the market over the last 3 months.
The Fox Corporation, especially their news station Faux News (did I spell that right?), has been rather controversial over the years. The network takes a unique perspective on many issues and features much more opinion based shows than the average news network.
Lets get a little more price data for the Fox Corporation tickers:
symbols = c('FOX', 'FOXA')fox = tq_get(symbols, from = '2017/01/01')fox %>%   ggplot(aes(date, adjusted))+  geom_line()+  geom_smooth(method = 'lm', alpha = 0.3)+  facet_wrap(~symbol)+  theme_classic()
It appears that over the last few years, both fox symbols have grown steadily
There was a huge drop in the price recently. and a continued downward trend following
Lets zoom in on recent prices:
tickers_df %>%   filter(symbol %in% c('FOX', 'FOXA')) %>%   ggplot(aes(date, adjusted))+  geom_point(alpha = 0.3, color = 'black')+  geom_line()+  facet_wrap(~symbol)+  theme_classic()
We can see that the price drop happened in early March. There was an initial large drop on March 13, followed by a brief rebound. Following the rebound the price, dropped and has continued trending downward.
Let’s turn to Google to figure out what happened on March 13. A quick Google search tells me that Media Matters organized a large protest against Fox Corporation on March 13th.
Following the March 13th drop, on March 19, 2019, Fox Corp began trading independently of 21st Century Fox. This made the price of the asset radically drop. This did not though represent weak performance by the asset, as one would initially think.
What actually happened is that Fox corporation began trading under 21st Century Fox’s tickers, 21st century Fox is now trading under the tickers TFCF and TFCFA. This led many news outlets to incorrectly report that FOX and FOXA were down 22%.
Finishing Up and Producing a Final Plot:
Alright now we want to share what we have found in an interactive plot, so that none technical users can have a fun, informative plot that they can interact with.
R has a lot of great frameworks for interactive data visualization. My Personal favorite interactive graphing library in R is the plotly library. Plotly is an interactive javascript library with APIs for many different programming languages. I am more familiar with the python API (and the python language in general).
With very similar code to as used in ggplot2 we can produce a beautiful interactive plot. I will embed the plot in this article.
p <- plot_ly(avg_return, x = ~avg_return, y = ~Volatility, type = 'scatter',        mode = 'text', text = ~symbol, textposition = 'middle right',        color = ~Indicator, colors = 'Set1',        textfont = list(size = 8)) %>%  layout(title = 'Average Return vs Volatility Over Last 3 Months In SP500',         xaxis = list(title = 'Averaage Return', zeroline = FALSE                      ),         yaxis = list(title = 'Volatility'                      ))
To recap, in this two part series we:
Went over an introduction to scraping, functional programming, and the tidyverse
Imported stock data for the whole SP500
Proposed a question, Which stocks had the highest average return over the last 3 months?
We answered this question, and by asking more questions of our data, we answered the question more fully and found what was ultimately interesting was the seemingly low average return and high volatility of the Fox Corporation tickers relative to the market.
This circular workflow of continuously asking questions of your data is common in data science
We need to be careful about data quality and make sure we understand our data. FOX, FOXA look like they are performing badly and DOW looks like it is performing way better than the rest of the market. These conclusions would be misleading!
Note from Towards Data Science’s editors: While we allow independent authors to publish articles in accordance with our rules and guidelines, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our Reader Terms for details.