Stream Stock Data
By: Ary Sharifian
Tags: Data Engineering, Data Science, Stock Market, Machine Learning, Python

Photo by Austin Distel on Unsplash

If you want to build some machine learning models to predict stock prices, what better way than by streaming live stock data. Using historical data is usually a good way to get started, but for the more “hardcore” data scientist/machine learning engineer trying to build the most robust models getting the right data is key. In this tutorial we will stream stock prices using the Python library robin_stocks. Since getting stock prices rarely is enough to build models, survey the other functions available in the robin_stocks library to build the “best” model. After this tutorial, you may consider building a bot with the model you built and see how much money it would gain (or lose). Now lets roll up our sleeves and get our keyboard dirty #dataengineering #python #stocks.

Outline:

Robin_Stocks API Configuration Provision Oracle Cloud Database (ATP) APEX RESTful Services Streaming Stock Prices Conclusion

Robin_Stocks API Configuration:

Robin_Stocks is a python library for interacting with the Robinhood API. Use of this library assumes you already have a Robinhood account. If so, you can use python scripts that can execute stock trades, get stock prices, and a variety of other functions inherent in your Robinhood application. Once you have installed the package using Pip or downloading the github, here are the scripts to log into your Robinhood account:

Once logged into your Robinhood account, you should notice the Json response including token type, expires in, scope, and a few other fields. The expires details the length of the session in seconds. My Json response states my session will be valid for 86,400 seconds which equals 1,440 minutes or 24 hours.

We can now use this library, we can start streaming stock prices:

The function above, expects user to enter a stock ticker in string format. The function then uses Robinhood to get the stock price for the ticker. Example of a valid stock_search value is “tsla” or “aapl”. The function returns the stock price in a float format.

Another important factor is time. Here is a function that returns the time when the stock price is collected:

Note: This is some additional documentation around how else to format the date time.

Provision Oracle Cloud Database (ATP)

To collect data, we use an Oracle Cloud Database which is preconfigured to transaction processing, Autonomous Transaction Processing (ATP). You can follow this guide to provision an instance of ATP. You can select the default values, and I encourage you to start with the free 20GB tier.

Note: if you are planning to stream only a few stocks and limited to the stock price, then a database might be overkill. You should just store the data in a pandas dataframe. But if you plan on building a more robust data model with data streaming from multiple stocks then a scalable cloud database is ideal.

APEX RESTful Services

We can use APEX to enable RESTful services for our ATP instance. This allows use to use POST requests to push data to our ATP database. I followed this blog to enable RESTful services.

Note: I only used the POST requests but the blog talks about both GET and POST requests.

You can test these services either through Postman or with Python scripts. Here are screenshots for my Postman test:

Note: make sure your database column types are consistent with the data you are attempting to post. If your POST request is successful but your data is not displaying in the database it could indicate that your database column datatypes are inconsistent/incompatible with your Json payload shown the Postman body above. Note: also, be sure to create a table in your database before attempting the post request.

Here is my database table column data types:

Note: if you are using ATP, you can use the SQL Developer Web services instead SQL Developer to check your tables. Simply click on your ATP instance > Service Console > Development > SQL Developer Web > Enter Credentials > Select Table.

Streaming Data:

Here are my scripts that complete the streaming process. First, we need the POST request url. This is the URL used in the Postman tests earlier. Next, we need to authenticate the request with our apex username and password.

Note: The APEX username is also the database schema. Make sure to login to the database using the same schema as the APEX username to check the stock table.

Finally, the poorly name function pull_push collects data from Robinhood using the get_stock_price function we defined earlier and POST/pushes the data into our ATP database. Notice that this function is applicable for one stock. To stream the entire stock market, you would need to rearchitect your functions and process. This tutorial is meant to provide a general overview for streaming stock data. I am returning the status_code for my POST script so that if there isn’t a 200 status code, the while loop stops. You want to run the pull_push function once before the while loop to ensure the proper 200 status code.