Full Stack Development Tutorial: Serving Trading Data with Serverless REST API running on AWS Lambda | by J Li | Towards Data Science

Serverless computing is a cloud-computing execution model in which the cloud provider runs the server, and dynamically manages the allocation of machine resources. Pricing is based on the actual amount of resources consumed by an application, rather than on pre-purchased units of capacity. — Wikipedia
(this post is also available in my blog)
This is the second post of my full stack development tutorial series. Previous post — Full Stack Development Tutorial: Visualize Trading Data on Angular SPA
Serverless computing shines in a way that it executes when it needs to. The price is totally based on the usage. There is no need for developers to maintain a server, patch security vulnerability, upgrade software etc.
Amazon’s AWS Lambda is an event-driven serverless computing platform. It has great runtime support including Node.js, Python, Java, Go, Ruby and C#. As an event-driven platform, Lambda is ideal for a scenario like handling change of events such as “file is being uploaded to S3”, “Database is updated”, “Sensor data is received in IoT” etc.
In this tutorial, we will leverage serverless framework to create a serverless REST API for our stock app in this tutorial. Serverless framework is an open-source framework that help develop and deploy serverless functions on multiple cloud environments including AWS Lambda.
Design our endpoint
Following the REST practice, lets define our endpoint /price?ticker=<ticker_name>. And response will look like the following to feed the chart in UI.
[  {    "name": "Ticker",    "series": [      {        "name": "2018-11-05",        "value": 1      },      {        "name": "2018-11-07",        "value": 1.0784432590958841      }    ]  }]
AWS
First you need an AWS account. Sign up if you don’t have an AWS account yet. Then create an IAM user. At the end you will get the credentials of IAM user. You will need it to set up the serverless cli, so that you can deploy and run your serverless service on AWS.
serverless config credentials --provider aws --key <ACCESS KEY ID> --secret <SECRET KEY>
Serverless installation
Follow instructions to install docker for deployment.
Create the serverless.yml file for the service. It needs plugin wsgi as web server to run the REST service. The built in flask development server isn’t performant enough for production. Note that the slim flag in pythonRequirements help to limit the size of the source code. There is size limit of source code for Lambda. The slim flag helps reduce the size. For that reason, it’s also critical to exclude all unnecessary files like the files in venv. The zip flag help zipping the source code before uploading to S3 for deployment. The rest of config is self-explanatory.
The deployment process is as follow in summary. Serverless framework builds and package the file as zip file. It then uses the IAM credentials to upload source code to S3 bucket (this bucket is created automatically by the serverless cli). Then it leverages CloudFormation to deploy the service the Lambda. All happens in once single command.
Implementation
The actual implementation is straightforward. When a request of historical data comes, it grabs data (csv file) from S3 and read it into a panda dataframe (S3 is an AWS storage service). Then we will iterate the dataframe to construct a response we defined above. As we define the Lambda entry point app.app in the serverless.yml. We will write our service in app.py.
Before running our application, download historical stock data from Yahoo finance and put it in a directory call data.
Then run start the application locally.
(venv) ➜  stock-backend python app.py* Serving Flask app "app" (lazy loading)* Environment: productionWARNING: This is a development server. Do not use it in a production deployment.Use a production WSGI server instead.* Debug mode: on* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)* Restarting with stat* Debugger is active!* Debugger PIN: 248-351-201
From now, you can verify if your REST endpoint works. Say if you have a SPY.csv file in data directory. Open browser and enter http://127.0.0.1:5000/price?ticker=SPY in the URL bar. The result of JSON response will be displayed in your browser.
[  {    "name": "SPY",     "series": [      {        "name": "2014-10-20",         "value": 190.300003      },       {        "name": "2014-10-21",         "value": 194.070007      },       {        "name": "2014-10-22",         "value": 192.690002      },       {        "name": "2014-10-23",         "value": 194.929993      }...]
Or you can use curl command to do the same test
curl http://127.0.0.1:5000/price?ticker=SPY
The above implementation relies on local storage to keep store the historical stock data. We can leverage AWS S3 to host the data. Go to the AWS Console and choose S3. Create a S3 bucket and give it a name. Make sure the name is globally unique.
Now, we can upload our csv file to the bucket.
Next, for Lambda to be able to access to this S3 bucket, we need to assign a S3 read only policy to the role of Lambda. Yes, this can be done via the serverless.yml config file without interacting with the AWS console. The updated provider in serverless.yml will look like follow
provider:  name: aws  runtime: python3.7  stage: prod  region: us-west-2  iamRoleStatements:    - Effect: 'Allow'      Action:        - 's3:ListBucket'      Resource: { 'Fn::Join': ['', ['arn:aws:s3:::<YOUR_S3_BUCKET_NAME>']] }    - Effect: 'Allow'      Action:        - 's3:getObject'      Resource:        Fn::Join:          - ''          - - 'arn:aws:s3:::<YOUR_S3_BUCKET_NAME>'            - '/*'
In the app.py, make sure we load data from S3 instead of reading from local disk.
# path = '{}/data/{}.csv'.format(os.path.dirname(os.path.realpath(__file__)), ticker)# df = pd.read_csv(path, index_col='Date', parse_dates=True, usecols=['Date', 'Close'],#                               na_values=['nan'])df = pd.read_csv('s3://<YOUR_S3_BUCKET_NAME>/{}.csv'.format(ticker), index_col='Date', parse_dates=True,            usecols=['Date', 'Close'], na_values=['nan'])
Now it comes to the exciting part. Let’s deploy our service to Lambda.
(venv) ➜  stock-backend sls deployServerless: Adding Python requirements helper.......Service Informationservice: stock-backendstage: prodregion: us-west-2stack: stock-backend-prodresources: 11api keys:Noneendpoints:ANY - https://9rn0mg6p6b.execute-api.us-west-2.amazonaws.com/prodANY - https://9rn0mg6p6b.execute-api.us-west-2.amazonaws.com/prod/{proxy+}functions:app: stock-backend-prod-applayers:NoneServerless: Run the "serverless" command to setup monitoring, troubleshooting and testing.
At the end, you will get the endpoint of your new API. This is the URL from API gateway. You can test it on your browser. Note that your URL will be different. Let’s use browser or curl to test. The following is the result. Now we have our backend service running on AWS Lambda. Deployment is easy.
If you’re interested in what AWS created, you can login your AWS console and find out. It creates S3 bucket and Cloudformation resources for deployment. A Lambda function is created with new IAM role for Lambda Execution. Also the role has user-defined role to access to the stock data S3 bucket.
I will connect the UI from previous post with the backend running on lambda, then deploy the UI to AWS. Stay tuned!
Recommnded reads:
Learn the A to Z of Amazon Web Services (AWS)
AWS Lambda in Action: Event-driven serverless applications 1st Edition
Amazon Web Services in Action 2nd Edition
Previous posts:
My posts about Finance and Tech
My posts about FAANG interview
From CRUD web app dev to SDE in voice assistant — My ongoing Journey to Machine Learning
Full Stack Development Tutorial: Integrate AWS Lambda Serverless Service into Angular SPA
Full Stack Development Tutorial: Visualize Trading Data on Angular SPA
Reinforcement Learning: Introduction to Q Learning