Crawling Stock Data with AWS
By: Yu Xiao
Tags: Aws S3, Aws Step Functions, AWS, Fun Projects, AWS Lambda

Introduction

This March, I was traveling in New Hampshire and spending decent chunk of my time chatting with my host Dave during dinners. He’s this kind of creative person who spends a lot of time day-dreaming. Once he mentioned this investment plan to me: “This is the biggest trouble ever hitting the US, so gold and silver prices will go up. I’m going to invest in small mining companies and they will grow 2, 5, 10 folds”

Then I decided I should try to dip my toe in the water of trading stocks. Although I didn’t expand my income by trading stocks, I do get interested in the price data generated day-by-day. Say if I want to analyze where the market is going everyday by myself, or I want to train and AI to help me make decisions, then I NEED DATA.

Today’s topic is about how to build a software to automatically fetch stock prices everyday and build a dataset from that. Here are some top-level requirements of the software:

able to fetch stock prices of current day for a set of pre-defined tickers able to store the fetched stock data for later use able to run on a pre-defined schedule: since stock market is only open Mon-Fri from 9am to 5pm

First Thoughts

Looks like the goal is pretty clear now, the next biggest problem then becomes: where to read stock prices? Google told me there are many public available API vendors which I can call to get stock data. Doesn’t the work look straight forward here? We can simply pipeline several AWS services together and looks like that should solving the problem. We can use an EventBridge rule to trigger events on a defined schedule, a Lambda Function can be written to call stock API’s, then we can just dump whatever we get to S3. This is something looks like the following diagram.

Backend Throttling Issue

However, things rarely go smoothly. After dive into public stock API’s a little bit, I realized one major constraint: throttling.

Throttling is ubiquitous, many web services rely on throttling to protect their availability against things like buggy client side code, huge traffic spikes, DDOS and more. Similarly, the stock API vendors use throttling to stop people from bringing down their servers by sending humongous amount of requests. For example, I chose to call Alpha Vantage for stock data, while it limits my traffic to 5 calls/minute. This means that I can’t call the api with the same token I have for more then 5 times in 1 minute. If I make extra calls, I will get no data but an error message saying I just breached the throttling limit.

OK, I see, so I can’t call you too fast or else I will get no data. That’s fine we can deal with it. To limit my TPM(transaction per minute, btw transaction is particularly misleading as we are not sending anything like money, but that’s sometime how people call API requests), I can simply implement wait or retry in my Lambda Function right? True, that will work, but there’s 1 issue that makes me want to think a bit deeper on my approach: cost.

Lambda Function is priced on memory-time(e.g. GB-millisecond). This means that AWS is counting how much memory your function use and how much time it takes to run. The issue with wait/retry is that the time I spend on waiting and retrying is counted into my bill. Say that my function takes 5 seconds to make the 5 calls I’m allowed, then I need to wait for 55 seconds until I can make another meaningful call. In this case, only 5 out of 60 seconds of that minute my function is bringing me value. In other words, I’m spending 12 times more money on my function.

After some napkin math, actually waiting in my Lambda Function is not a big concern on my scale, it will roughly takes my $2 to crawl 2,000 tickers per month. However, I still want to think a bit more on how I can improve this. Why? It looks like a fun puzzle to solve.

Service Orchestration with AWS Step Function

Let me summarize the requirement posed by backend API throttling:

the software only makes 5 API calls per lambda function invocation the software is able to wait for ~60s between lambda invocations

One way to approach these requirements is to use EventBridge cron rule to trigger lambda function every minute. The biggest issue for this approach is we can’t easily specify inputs for each lambda invocation, we may need a DB table to store information like which tickers are already crawled and which are not.

The approach I want to share here is the one leverages AWS Step Function. The one-liner understanding of Step Function from me: Step Function is a service helps us to build finite state machine used to orchestrate other AWS services for application logic’s needs. In Step Function, a state machine is defined using ASL(Amazon States Language). Several relative core concepts are:

State: matches common understanding of state machine, we specify what kind of work the app need to do in a state. For example, in a state we can invoke a Lambda Function(Task state), branching based on outputs from previous state(Choice state), wait for a defined amount of time(Wait state), etc. Transition: we also need to specify how our app’s states flow by defining transitions. These can be pre-defined or conditional transitions.

The magic here is Step Function offers a state called “Map” state, which allows us to input a list of items and loop through each item and use it to trigger a “inner” state machine. For example, let’s say we specify 2 batches of tickers we would like to crawl in the following way:

{"batches": [ {"batch": ["AMZN", "MSFT", "GOOG", "BABA", "XOM"]}, {"batch": ["WORK", "NOC", "PFE", "BA", "NVDA"]}]}

Map state allows us to loop through each “batch” in the above input through a state machine. In our case this state machine will consist of 2 state: “Crawl” and “Wait”. In this way, the state machine will look like the following

In plain English, the app flow is this:

LoadTickers state(Type: Task): load batches of tickers from S3(a json file looks like the above sample) RunCrawlers state(Type: Map): split the loaded tickers into individual batches, pass each batch into the following state Crawl state(Type: Task): given 1 batch of tickers, make API calls to Alpha Vantage and store results in S3 Wait state(Type: Wait): wait for 60 seconds

The end results of the above example looks like the following:

Summary

AWS Step Function is a useful service to service integration if you have multiple components and tricky dependencies in your app. AWS console provides useful visualization and trouble shooting tools for Step Function to developers. My experience of building a state machine with console is pretty seamless. In summary it’s a pretty good learning for me.