Automating data collection with Python on GCP | Towards Data Science

This article will show you one of the ways you can host a Python script on Google Cloud Platform. Learning to do so can be especially useful for anyone who wishes to automate the process of collecting financial data from APIs, and is looking for an efficient and secure way to store the whole process on a cloud platform.
I will illustrate this process by collecting API data on the daily ingestion of stock price for a given sample of firms. The data can then be used to conduct financial modelling.
The first step is to gather the list of Ticker Symbols (the stock identifier for each security) for the relevant list of companies; in this case, I am using the SP500 Index components, whose list is readily available online at the relevant Wikipedia page.
In many cases the list of ticker symbols may already be present in csv/excel format and automatically downloaded from other sources; in this case, please skip ahead to step 2.
From the table, you can identify the Symbol corresponding to each security, which you will use later in order to fetch stock price data from the API. This is a useful table to keep as reference in the target database as you can then periodically run the script to collect data from this table and obtain the latest view on the SP500 members. Given the market capitalization of these companies, changes are infrequent, but it is good practice to update this periodically to account for firms entering/exiting the Index. Once a quarter could be a considered a good update frequency for this kind of use case.
In the below steps the script will be executed monthly, but the target frequency should be adapted to fit the task you are looking to accomplish.
To gather the table of member firms, you will need to write a Python script. The requests and Beautiful Soup Python packages are good options for scraping the Wikipedia Page and extracting the table element from its HTML code.
Let’s now go through the relevant Python code sections to put in place within the script in order to achieve this.
Import relevant Packages: As a first step , you will need to import the relevant Python packages and define the url variable, which points to the online data source. You should also import the pandas package (“import pandas as pd”) as you will need it later on.
Import relevant Packages: As a first step , you will need to import the relevant Python packages and define the url variable, which points to the online data source. You should also import the pandas package (“import pandas as pd”) as you will need it later on.
import requestsfrom bs4 import BeautifulSoupimport pandas as pdurl="https://en.wikipedia.org/wiki/List_of_S%26P_500_companies"
2. Scrape data: You then need to fetch the relevant url content with the request.get method. Then, you can extract its text with the .text method, and finally parse it into a Beatiful Soup object.
r = requests.get(url,timeout=2.5)r_html = r.textsoup = BeautifulSoup(r_html, 'html.parser')components_table = soup.find_all(id="constituents")headers_html = soup.find_all("th")df_columns=[item.text.rstrip("\n") for item in headers_html]
In order to locate the table within the HTML, you can inspect the code of the page (this can be done by clicking Ctrl + Shift + I on Google Chrome) and see that the table element you are interested in has an ID of constituents, which you can use to store table contents into the components_table variable. You can then use .find_all to locate all elements in a page with a given ID. In this specific page, there are two tables with that ID, referenced by the components_table object. Given this, we will only work with the first one, which we specifically select at step 3.
You can also use the same method (soup.find_all) to identify table headers, which you will use as column names in our dataset, and store them into a list variable, named df_columns.
3. Store data into a Pandas Dataframe: next, you need to isolate the first table by indexing the components_table object (which resides at index 0), and then find all the rows within the body of the table, which are not headers; go ahead and store the results into data_rows.
Then you can iterate over the rows, split the text content of every row into a list element, filter for null values and store it into a list of name stock, which you will have to then append into a list of list named rows.
You can then turn rows into a pandas DataFrame object, parsing component_headers as column names. Component_headers is nothing but a sublist of df_columns, which indentifies the columns pertaining to the first of the two tables with the ID of constituents.
components_headers=df_columns[:9]data_rows=components_table[0].find("tbody").find_all("tr")[1:]rows=[]for row in range(len(data_rows)):        stock=list(filter(None,data_rows[row].text.split("\n")))    rows.append(stock)S_P_500_stocks=pd.DataFrame(rows,columns=components_headers)
4. Save dataset into csv file: as shown in the code below, you will then remove the redundant column named SEC Filings, and use the .to_csv method to save the dataset locally. You can now use this table to fetch stock price data from the API.
S_P_500_stocks.drop("SEC filings",inplace=True,axis=1)S_P_500_stocks.to_csv(r"/home/edo_romani1/my-python-directory/SP500stocks.csv",index=False)
Tiingo is a financial markets API which provides a variety of information relating to publicly-listed companies over several markets; in this example, we are interested in collecting the daily historical stock price information (Date, Open, Close, High, Low, Volume) and will therefore refer to the related Tiingo API documentation.
Let’s break the code down step by step and see how we can gather 20+ years of daily stock price information for each of the SP500 member firms.
1.Import relevant Packages: As before, proceed to import the necessary packages. This time, you’ll additionally need to import the datetime module, as you will use later on to define the time window along which we want to extract data from the API.
import requestsimport pandas as pdimport datetimefrom datetime import datetime,timedelta
2. Define API Token: to interact with the API, you need to get an API Key, or token, in order to identify ourselves as registered Tiingo API users. With Tiingo, you can easily set up an account and get your token. Once done, all you need to do is declare it in the code and set it as a variable, such as the illustrative example below.
token1="123RUXBJKSJKFJD23Y348"
3. Import the SP500 members table yousaved as a csv file in Step 1; this can be easily done with the .read_csv method in pandas.
SP=pd.read_csv(r"/home/edo_romani1/my-python-repository/SP500stocks.csv")
4. Call the Tiingo API and get historical stock price data
You first have to set up a list of the ticker symbols to iterate over in order to make an API call for each of them, storing them in a variable ticker_symbols, which you can extend with the addition of the SP500 Index value (“SPY”) in the following line.
You can also set up the latest day (end_date variable) for which you want to call data from the API, as for each stock you will be calling the stock price history and so a time window to partition by must be set. Set the end_date to yesterday (this is calculated below with the use of the datetime and timedelta modules — datetime.now.date() gets you today’s date, to which you subtract 1 day using the timedelta function, in order to get today-1 days= yesterday).
Note: Tiingo has hourly and daily API call limits which were handled with the usage of the rate_limiter package used as a context manager. This is important to know if you are making many calls per hour/day.
Lastly, you need to initialize an empty list data1, which you will use to store the historical dataset for each stock price / API call.
ticker_symbols=list(SP["Symbol"])ticker_symbols.append("SPY")end_date=str(datetime.now().date()-timedelta(days=1))
The next lines of code are the ones where you will actually call the Tiingo API to get data for each ticker symbol of interest. First, you need to set the symbol, end_date and token parameter in the Tiingo url to submit into the request.get method; you then need to store the response into a variable r, which corresponds to the csv file containing the stock price history for the given ticker. The following few lines in the code below convert such values into a list then a dataframe df, which you will then append to the data1 collection before moving on to the next ticker_symbol in iterative fashion.
Note: the try clause is used to take handle timeout errors caused by the requests package, the following except clause repeated the same code with an extended timeout time to handle longer responses; the code is equal to the below with the only difference being the value of the timeout parameter (which was doubled), and was thus left out for simplicity.
data1=[]for symbol in ticker_symbols:        #with rate_limiter:                try:                        url = "https://api.tiingo.com/tiingo/daily/{}/prices?startDate=2000-01-03&endDate={}&format=csv&token={}".format(symbol,end_date,token1)                        r = requests.get(url,timeout=10)                        rows=r.text.split("\n")[1:-1]                        df=pd.DataFrame(rows)                                                                        df=df.iloc[:,0].str.split(",",13,expand=True)                        df["Symbol"]=symbol                        data1.append(df)
5. Save dataset into csv file: As a final step, you can use the .concat method to concatenate the list of dataframes into a single final dataset, before finally saving it into a csv file like before.
df_final=pd.concat(data1)df_final.drop(df_final.iloc[:,6:13],axis=1,inplace=True)df_final.to_csv(r"/home/edo_romani1/my-python-repository/historicalSP_quotes.csv",index=False)
Below is the csv view from the final output, where we can observe the initial date and the relative price data for the first firm in the sample.
Now that you have two fully functioning Python scripts which get stock data from the Tiingo API, let’s see how you can automate their running with the use of the Google Cloud Platform (GCP), so that every day in which the market’s open you can gather the latest quotes of the prior day. I will display how we can make use of Google’s Compute Engine, Cloud Functions, Cloud Storage and GoogleBigQuery to set up a simple data ingestion workflow which will be based off of these two scripts running on the cloud, and thus moving away from our locale machines.
Instead of running our two Python scripts locally, you can launch a VM instance using GoogleCloud ComputeEngine. You can easily start up a Linux VM and connect to it via SSH.
Once connected, you will be presented with the following terminal view:
At this point, using the settings button on the top right, you can select “Upload file” to swiftly move both our Python scripts within the VM’s file system.
Once this is done, your scripts will be stored on the VM. Do not forget to change local path references in the python scripts, when the datasets are saved as csv files; on the terminal, you can set up a new directory with the mkdir command and get the file path for a given file with the pwd command. More detailed info can be found here. You can launch and run any of the two scripts from the terminal using the python (or python3, depending on your python version) command followed by the location of the py file to be executed.
As a start, you would want to run the SP500 members collection script first so that the second script can correctly import the csv table from which to extract the list of ticker symbols.
Finally, you can automate the execution of the two scripts directly from the linux terminal using cron jobs; you can access the crontab with the crontab -e command and then set up cron job schedules followed by the python file execution.
In this setup, I am running the first script once a month, in order to update the SP500 components, and the stock price API script every day in which the stock market is open, so every Tuesday to Saturday (in order to get the last day’s worth of data from Monday to Friday of each week). A sample view of the described crontab is included below.
Now that you have automated the execution of the scripts, the next step is to move both output tables (the SP500 members table and the stock price data table for all members) to a storage location in GoogleCloud for reference. I have used GoogleCloud Storage to do so, and will show how you can stage data in GoogleCloudStorage before moving it to a data warehousing solution (which in this case will be GoogleBigQuery).
Before you do so, you can quickly create two different storage buckets for your respective files using GoogleCloud’s Console. Once created, you can simply upload your two csv output tables to each bucket.
To communicate with GoogleCloud Storage you can use the python CloudStorage client library which is available with GoogleCloud; do not forget to set up authentication credentials in order to communicate with your GoogleCloud account resources.
After setup, you only need to add the following code blocks to the end of both of our Python scripts in order to complete the process:
import osfrom google.cloud.storage.blob import Blobos.environ['GOOGLE_APPLICATION_CREDENTIALS']=r"/home/edo_romani1/My First Project-69347b3b5af2.json"storage_client = storage.Client()bucket = storage_client.get_bucket("spcomponents")data=bucket.blob("SP500stocks.csv")data.upload_from_filename(r"/home/edo_romani1/my-python-directory/SP500stocks.csv")
The first two lines import the necessary packages; then the code sets the application credendentials via os.environ , which points the location (on the Linux VM) of the json file which contains our GoogleCloud individual credentials.
You can then initialize the storage client, get bucket information with the .get_bucket method, select the object which you would like to update with bucket.blob, and then update the object every time the python script runs with the .upload_from_filename command, which points to the Linux VM location of the relevant file. The example above is showing the process for the first output table only, and you only need to change the necessary file bucket/path/names/location details in order to replicate the process for the stock price dataset.
Now, for every time each of the python scripts is run, the respective bucket on GoogleCloudStorage is updated with the latest data.
Cloud Functions allow us to run event-driven code which targets specific GoogleCloud resources, such as, in our case, GoogleCloud Storage buckets.
Let’s see how you can set one up which does the following task: every time your csv output tables are uploaded into GoogleCloudStorage, you can move and upload such tables into GoogleBigQuery, which is GoogleCloud’s data warehousing solution (one of the many) and quite ideal for working with relational data such as those in this tutorial.
Create a GoogleBigQuery dataset and tables as a necessary setup step. You can set GoogleCloudStorage as the table source for each of our two tables and temporarily load our latest available tables.Write code which uploads table data from GoogleCloudStorage into GoogleCloudBigQuery; this will be the actual code that the CloudFunction will run every time an update on GoogleCloudStorage will be generated. The code for this table is the following:
Create a GoogleBigQuery dataset and tables as a necessary setup step. You can set GoogleCloudStorage as the table source for each of our two tables and temporarily load our latest available tables.
Write code which uploads table data from GoogleCloudStorage into GoogleCloudBigQuery; this will be the actual code that the CloudFunction will run every time an update on GoogleCloudStorage will be generated. The code for this table is the following:
def csvloader(data,context):    from google.cloud import bigquery    client = bigquery.Client()    table_ref = client.dataset("csm").table("SPhistorical")    job_config = bigquery.LoadJobConfig()    job_config.write_disposition =     bigquery.WriteDisposition.WRITE_TRUNCATE    job_config.skip_leading_rows = 1        job_config.source_format = bigquery.SourceFormat.CSV    uri = "gs://csm_erat/Data/historicalSP_quotes.csv"    load_job = client.load_table_from_uri(uri, table_ref, job_config=job_config)load_job.result()    destination_table = client.get_table(table_ref)
The code defines a csv_loader function, which imports the necessary BigQuery client package, initializes it as client, connects to the previously created BigQuery table (this example is shown only for the historical stock price data, but the structure is identical for each file we want to move) as table_ref. Then, the code sets up a load job with certain configuration parameters in the following lines of code (such as indication of overwriting present BigQuery tables as the new table is loaded, indication of the n number of rows to skip on read and the source file format -csv-). It then defines the source path (uri) in which to find the new data to upload, and loads it with the .load_table_from_uri method and relevant parameters. The uri points the the GoogleCloudStorage location of the data. The data and context parameters have to do with the event payload and the metadata triggering the event, but are not part of the core load job in this regard. More info can be found here.
Full details on how to use the BigQuery client library can be found here. It also contains helpful information on how to setup authentication.
3. Setup a Google CloudFunction with the code defined at step 2
You can finally deploy our Cloud Function right from the GCP Console, by clicking on the Cloud Functions resource and clicking on “Create Function”; you can then enter the name of the function, the trigger (Cloud Storage in our case) and the runtime (Python3). Once done, you can paste in your function code and deploy it; the function will then be up and running, and you will be able to monitor it via event logs available within the GCP Console.
Step 6: View the dataset and tables in BigQuery
You can now access the latest data pull from the API directly within GoogleBigQuery, by accessing its UI, selecting our GCP project > dataset > table > Preview. You can also check latest update time on the details section.
Your workflow is officially set up and it only takes 5 minutes to run from start to end.
This a quite simple solution you can set up with just a couple of Python scripts and GCP’s free tier account.
A nice next step would be to call the entire history of stock price data only once, and modify the API call (within the url variable) to fetch only yesterday’s values.
Hope this step-by-step guide will be useful in setting up your next Python project using financial data on GCP!
towardsdatascience.com
Access my free Data Science resource checklist here