Data Pipeline in GCP: Cloud Function Basics | by Mostafa Varzaneh | Towards Data Science

Most Data Scientists, prefer to own the end to end data pipeline of their models, but owning a pipeline requires a lot of engineering effort.
In this article, I will talk about Cloud Function, which is a serverless, easy, and cost-effective option.
GCP provides a simple scheduling tool called “Cloud Scheduler”. From the navigation menu on the top left find Cloud Scheduler.
Give your job a name and description. Next, you need to provide the frequency of the schedule. If you ever worked with any scheduler including Cron Jobs, you will be fine here.
Select time zone and Pub/Sub as the target.
Once you click on Pub/Sub options for Topic and Payload appears. The topic name will be used later as the name of your Python function so keep in mind to use a topic that is acceptable as a function name in Python. Give payload some arbitrary string for now.
If you prefer to use the command line you can run the following:
gcloud alpha scheduler jobs create pubsub scheduler-name --schedule="0 8 * * *" --topic="my_pipeline_topic" --message-body=" "
You should now be able to see your scheduled job in Cloud Scheduler.
I recommend you to read Google's documentation about Pub/Sub. But in a brief sentence, Pub/Sub allows you to send messages between different applications. You can leverage Pub/Sub for batch and stream data pipelines.
Now use the topic to create a Pub/Sub topic
gcloud pubsub topics create my_pipeline_name
You have the option to create the Pub/Sub topic using UI:
You have the option to create your Cloud Function from UI but I do not recommend it. First, if there is a bug in your code you will lose your work. Second, you can easily lose track of your changes and versions.
As the first effort, let's create our functions locally and deploy them manually to GCP.
What do you need?
you need a “main.py” function. This is the wrapper around all your functions and classes. Inside main.py there should be a function with our Cloud Function name.
your imports heredef my_pipeline_name(event,contex):    your_code_here
Keep the rest of your functions in the same folder, it is easier!
Here is the definition of event and context according to Google:
event (dict):  The dictionary with data specific to this type of         event. The `data` field contains the PubsubMessage message. context (google.cloud.functions.Context): The Cloud Functions event         `timestamp` field contains the publish time.
So leave them!
Create a .gcloudignore file. This file is like a .gitignore but for cloud deployment. You may include the files you do not want to deploy (like secret keys which you kept in your folder for local tests)
Create a requirements.txt file with the required packages.
during your run, you may need to save some files, but the root of your cloud function is read-only! You need to use /temp/ directory for your temporary and modifiable files.
It is time to deploy!
gcloud functions deploy my_pipeline_name  --runtime python37--trigger-topic my_pipeline_topic --timeout 540
Here is a big limitation! The maximum timeout for cloud function is 9 minutes according to the documentation.
Check Zdenko’s article and GitHub for more details.
Go to Cloud Functions on GCP UI and then click on the function name, it will redirect you to a new page called “function details” which has the latest deploy version
In the general tab, you can find Invocation, Execution time, Memory usage, and Active instances. All of these are important factors for billing (I know Cloud Functions are cheap but it is good practice to keep track of the cost). You can refer you Romin’s Article about Cloud Function pricing.
You can click on edit on top middle and change memory, functions, or ...
Clicking on Source, you will find the deployed files and you can click on edit to modify them online (not recommended).
Go to the Testing tab and click on TEST THE FUNCTION the code will run and returns an error if there is any problem. But this is a test for Cloud Functions, what about Pub/Sub and Scheduler?
You can go to “Cloud Scheduler” from Navigation Menu and you will find a Run Now button in front of your job name. You may use it for testing and check the message under theResult column.
I am not comfortable with deploying my secret key with the code, here I will review my top two options for handling secrets.
The first one is to add environment variables to your Cloud Functions. You can either use UI or command line. This option is not my favorite since first the secret is sitting in the UI and second I have to deploy it for other functions again.
The second option is to use the secret manager on GCP. I will borrow some material from Dustin’s post on managing secrets in Cloud Functions.
Now you can use cloud shell. The icon is on top right next to the search bar.
Enable “Secret Manager API”:
gcloud services enable secretmanager.googleapis.com 
Create a secret:
echo -n "your secret text here" | \    gcloud beta secrets create my-secret \      --data-file=- \      --replication-policy automatic
If you have double quotations in your secret file, use single quotations around it.
Write a function to retrieve your secrets:
import osfrom google.cloud import secretmanagerimport jsondef get_secret(secret_name):    client = secretmanager.SecretManagerServiceClient()    project_id = 'my_gcp_project'    resource_name = f"projects/{project_id}/secrets/{secret_name}/versions/latest"    response = client.access_secret_version(resource_name)    secret_string = response.payload.data.decode('UTF-8')    secret_string = json.loads(secret_string)    return secret_string
You can call this function in the rest of your codes to retrieve your keys. Make sure to add “google-cloud-secret-manager” to your requirements file.
You are set! Go to the UI and test!
Even though you can deploy from your local, Google provides an option for deploying from a source control like GitHub or BitBucket. Let’s go through it step by step.
Mirror you repository
In the Google Cloud Console, open Cloud Source Repositories.Open Cloud Source RepositoriesClick Add repository.The Add a repository page opens.Select Connect external repository and click Continue.The Connect external repository page opens.In the Project drop-down list, select the Google Cloud project to which the mirrored repository belongs.In the Git provider drop-down list, select Bitbucket.Select the checkbox to authorize Cloud Source Repositories to store your credentials.Click Connect to Bitbucket.Sign in to Bitbucket with your machine user credentials.Click Authorize GoogleCloudPlatform.This option grants Google Cloud read access to your repository.When authorization finishes, you’re returned to the Connect external repository page. A list of repositories opens.From the list of repositories, select the repository you want to mirror.Click Connect Selected Repository.
In the Google Cloud Console, open Cloud Source Repositories.
Open Cloud Source Repositories
Click Add repository.
The Add a repository page opens.
Select Connect external repository and click Continue.
The Connect external repository page opens.
In the Project drop-down list, select the Google Cloud project to which the mirrored repository belongs.
In the Git provider drop-down list, select Bitbucket.
Select the checkbox to authorize Cloud Source Repositories to store your credentials.
Click Connect to Bitbucket.
Sign in to Bitbucket with your machine user credentials.
Click Authorize GoogleCloudPlatform.
This option grants Google Cloud read access to your repository.
When authorization finishes, you’re returned to the Connect external repository page. A list of repositories opens.
From the list of repositories, select the repository you want to mirror.
Click Connect Selected Repository.
On the top left below “Cloud Source Repositories”, you can find the name of your repo and also the active branch. Click on the drop-down menu and change see the rest of the branches.
Deploy From Cloud Repo
You can deploy to Cloud Function from Cloud Source Repository using the following commands:
gcloud functions deploy my_pipeline_name \  --source https://source.developers.google.com/projects/[gcp_project_name]/repos/[cloud_source_repository_root]/moveable-aliases/[branch]/paths/[folder_under_that_branch]/ \  --runtime python37 \  --trigger-topic my_pipeline_topic --timeout 540
Now check your could function, the version should be incremented. Go to theTESTING tab and enjoy passing the tests!
Now that you mirrored your repository to “Cloud Source Repositories” if you check the source of your cloud function you see that it points to your repo now!
This step may seem redundant since you can deploy from your local, but let's go to the next step.
So far we learned how to deploy from local and from cloud repo, let’s learn setting up a CI/CD!
You need to add a “cloudbuild.yaml” to your directory and put the following content:
steps:- name: gcr.io/cloud-builders/gcloud  args:    - functions    - deploy    - my_pipeline_name    - --source=https://source.developers.google.com/projects/[gcp_project_name]/repos/[cloud_source_repository_root]/moveable-aliases/[branch]/paths/google_ads/[folder_under_that_branch]/    - --trigger-topic=my_pipeline_topic    - --runtime=python37    - --timeout=540
Now from GCP’s Navigation Menu go to cloud build, go to Triggers and click on “create new”. It should be straight forward now!Give your trigger a name and description. I want to trigger the build every time I push to the branch, so will select that option. From the source section, I will find the mirrored repo from step 6 and enter the name of the branch.
Since the build configuration is saved in the cloudbuild, I will select the first option and copy and paste the path to the YAML file.
You are all set, click on create and enjoy your CI/CD pipeline!