How to use AWS Lambda and CloudWatch for beginners | by Denny Asarias Palinggi | Towards Data Science

I found a cool website (https://covid19api.com/) where we can easily access COVID19 data using free API. This gave me an idea to create simple function to grab the data using AWS Lambda and save it to S3. The script will be executed daily automatically using CloudWatch.
Below is the list of tasks that I needed to follow:
Write a function to retrieve the data and save it to S3.Deploy the function as an AWS Lambda function.Create an event on AWS CloudWatch to run the function on schedule.
Write a function to retrieve the data and save it to S3.
Deploy the function as an AWS Lambda function.
Create an event on AWS CloudWatch to run the function on schedule.
So, let’s do it!
Write a function to retrieve the data and save it to S3.
Write a function to retrieve the data and save it to S3.
In the beginning, I wrote the code to retrieve the data on my local computer. Here are both the code and the result:
import requestsimport pandas as pdfrom pandas.io.json import json_normalizebaseUrl = "https://api.covid19api.com/total/dayone/country/indonesia/status/confirmed"response = requests.get(baseUrl)df = json_normalize(response.json())df['change (%)'] = df['Cases'].pct_change()*100
As you can see, the script is working just fine. Next, we need to put the script on the handler function, which is the function that Lambda will execute.
import requestsimport pandas as pdfrom pandas import json_normalizeimport boto3from datetime import datetimefrom io import StringIO def lambda_handler(event, context):        baseUrl = "https://api.covid19api.com/total/dayone/country/indonesia/status/confirmed"    response = requests.get(baseUrl)    df = json_normalize(response.json())    df['change (%)'] = df['Cases'].pct_change()*100            bucketname = 'my-beautiful-bucket'    filename = 'corona_dataset.csv'        csv_buffer = StringIO()    df.to_csv(csv_buffer)        client = boto3.client('s3')        response = client.put_object(        ACL = 'private',        Body= csv_buffer.getvalue(),        Bucket= bucketname,        Key= filename    )
I added two new libraries which are boto3 and StringIo, I also added scripts to convert the dataframe into csv and saved it into my s3 bucket called ‘my-beautiful-bucket’. Copy this script and save it into a .py file called lambda_function.
2. Deploy the function as an AWS Lambda function.
Since AWS Lambda does not include external python libraries, we can’t just solely deploy lambda_function.py. We need to compress lambda_function.py and all those libraries into one .zip file.
So the next thing to do is to create a new directory and put the lambda_function.py file into this new directory. For windows users, open Command Prompt and go to the directory that we just created, Install all libraries to local directory using pip:
pip install -t . pandas
Now there’s a tricky part since AWS Lambda uses Amazon Linux as the operating system. Therefore, we need Pandas and Numpy which are compatible with Linux.
To solve this issue, first remove Remove pandas, numpy, *.dist-info and __pycache__ . Next is to open https://pypi.org/project/pandas/#files and download the latest*manylinux1_x86_64.whl package. The same thing with Numpy: open https://pypi.org/project/numpy/#files and download the latest Linux compatible package. Extract both .whl files in our directory and compress all of it in a new .zp file (Note: Don’t forget to remove whl files, *.dist-info, and __pycache__.):
Now it’s time to create a Lambda function, log in to your AWS Account, find the Lambda service, open it, and create a function.
Next, go inside the function, choose “Upload a .zip file” as a Code entry type, click the upload button, choose the .zip file and click the save button to deploy it.
If the size of the .zip file is too large, we can just upload it to our S3 bucket and upload the .zip file from Amazon S3. Click the test button to run the script.
3. Create an event on AWS CloudWatch to run the function on schedule.
Let’s say that I want my function to run every day at 10:00 GMT. Go to CloudWatch Service, click Rules and click Create Rule Button. Choose Schedule and then choose Cron expression instead of Fixed rate, then type 0 10 * * ? 0 10 * * ? * in the text box which means that it will run every day at 10:00 GMT. Choose our Lambda function, click the configure details button, type the name of the event, and finally click the create rule button to create the event.
Hopefully, everyone who reads this article will find it useful. Of course, there are other AWS services that you guys can include in this project to make it even better, such as AWS Athena to analyze the data using SQL language in S3 and AWS QuickSight to create and analyze visualizations of the data. Feel free to explore!