Stock Market Data Collection & Feature Engineering Using Python | by Posey | Towards Data Science

Finding appropriate data and pre-processing that data is arguably the most important part of all modern machine learning problems. In an age of unlimited libraries, APIs, and guides for how to build and implement optimal models, the actual data is overlooked. The advice is usually as simple as, “just find some data” or even better yet “go collect some relevant data.” This is of course rubbish. Collecting data is hard; collecting useful data is even harder. Some datasets are the result of years of data collection. In the case of financial data sets it is the result of decades of data accumulation. Most examples you’ll find on the web include pre-cooked datasets from Kaggle, the infamous Iris data set, housing, etc. What separates a great AI company from just an okay company is not usually the quality of models but rather the quantity and quality of data.
It’s hard to make predictions, especially about the future...
Niels Bohr
It is overwhelming nowadays to try and compete with established giants in AI and machine learning. How on Earth can we as normal people with average budgets go about acquiring the data necessary to engineer useful systems? Our models can be of similar quality to the big players, but our data is usually lacking. In this article we examine how to level the playing field. This information can be utilized for all sorts of problems, not just machine learning for the stock market. But we approach the problem in this article from a stock selection perspective.
This is how the average workflow looks, more or less. Arguably the most important parts are 1–3. The performance of our model oftentimes is entirely dependent upon the quantity and quality of our data. #2 defines the initial quantity and quality of our data. #3 defines making our data machine-readable, i.e. usable by our model as well as cleaning the data and squeezing as much utility out of the data as possible. There are all sorts of slick methods we can use to perform this processing which we’ll discuss in more depth later on in the article.
Quite clearly the problem we are looking to approach is stock selection. How can one value a stock? We’ve discussed such valuations in the past so we will not go too far in depth. If we are trying to develop a model that is capable of providing a market view, it would help to have some understanding of the markets and what data will be valuable to the model. In the case of deep learning we can potentially avoid this problem. With deep learning, if we gather enough data it’s possible for our model to be very useful without much knowledge of what is “good data” if we gather enough quantity of data. Gathering the quantity of data to make this a reality can be very difficult. So we will be focusing on finding quality data and processing the data well.
This is the part where most people get stuck. How on Earth can I find useful data when the big firms are tapped directly into all of the world’s top data sources and are accessing data at unprecedented scale? How can I find and collect this data on my personal computer? I contend that the average individual has the resources to gather sufficient data to build useful models. Storage these days is relatively cheap. And there are tons of data sources available that most people just aren’t aware of. Some of these data sources come from academia, some from random APIs, amazing open source people who provide this data for free, governments, etc.
Here are some useful data resources:
Favorites: https://db.nomics.world/ This resource is especially helpful. It’s a compilation of massive amounts of economics datasets from all over the world. You can select as many data sets as you want → add them to your cart → download all datasets with one click
https://github.com/addisonlynch/iexfinance This resource is my go-to for gathering data on stock prices, balance sheets, analyst predictions, and all sorts of great information.
Census Data: https://www.census.gov/data.html
Useful Collection of Forecasts: http://www.forecasts.org/stpoor.htm
Massive Collection of Indicators: https://www.assetmacro.com/
Labor Statistics: https://www.bls.gov/
Data from the US Treasury: https://home.treasury.gov/
Since we’re using Python, our first step is probably to read in the data from our various CSV files. One should look to merge all of this data together into a single dataframe and perform some exploratory analysis.
Cleaning Data
Inevitably, some of our data will have holes in it, some will just be garbage and completely out of left field, and some can be made more effective by scaling or normalizing. We can also add artificial data to our dataset if we need more data to train the model. This is especially popular right now in image processing where companies like Nvidia are simulating environments to help train CNNs.
The simplest route to take when preparing data that includes missing data or invalid data is to use Pandas built-in functions like dropna().
df.dropna(inplace=True)
dropna() will remove all NaN entries from our dataframe.
df.fillna(df.mean(), inplace=True)
fillna() will replace NaN entries with the desired argument. In our example above we use the column mean as a replacement. There are tons of possible approaches. Another possible approach is to take the average of the nearest two entries to that NaN entry. Some of these approaches are well-defined in various material on information theory.
Feature Engineering
Oftentimes it can be useful to identify what feature combinations will be most useful for training our model. These are going to be features that are highly separable. To that end, a visualization can be especially useful.
We can use the library seaborn to automatically build a visualization with our features vs. our output.
sns.pairplot(data=df, hue="asset_price")
These distributions aren’t terribly separable. We can see the first feature is probably more separable than the second feature. In a deep learning model we probably just throw every bit of data into the mix. But in some non-deep learning applications we can get better performance out of just training on the data that is more linearly separable.
We can also score features under the same principle in a more quantitative fashion. We use Scikit-Learn’s SelectKBest to score our features.
Obviously the higher the score the more impact the feature has on our output according to the SelectKBest algorithm. In our case, we use the Chi-squared test for our scoring:
The details aren’t overly important, just know your score indicates “goodness of fit.” Here is the implementation:
Scaling Data
There are many approaches to this part of the processing game. One of the most useful functions provided by Scikit-Learn is scale(). This uses normalization to bring all of the data into a more digestible form. You can read more about scaling here. In this case we also flatten our output variable since 1-D is our desired dimension:
Let’s continue the conversation on Twitter!
To support my writing and get full access to all articles on Medium, visit https://posey.medium.com/membership
I hope you learned something new, maybe a new method or a fresh approach. Many of these methods likely already exist in your workflow, but some may not and you should try adding them to your toolkit and seeing what works best. If you’re struggling to get very great results out of your models, I recommend spending more time on your data and less time trying to find a model that magically fits your data; you’re likely just overfitting. The majority of failing models occur because of a lack of data, low quality data, or just mismanagement of data. Data, data, data...
Note from Towards Data Science’s editors: While we allow independent authors to publish articles in accordance with our rules and guidelines, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our Reader Terms for details.