Review apps feature on Google Kubernetes Engine using Jenkins
By: Anil Saravade
Tags: Nginx Ingress, Jenkins, Heroku, Docker, Google Kubernetes Engine

3.3. Configure a user for GitHub webhook

To create a new user in Jenkins click on Manage Jenkins → Manage User → Create User Complete the details.

Assign permissions to git user.

1] Go to Manage Jenkins → Configure Global Security → select Enable security.

2] Under the Authorization section, select the Project-based Matrix Authorization Strategy.

3] Add the git user and assign the below permissions.

Overall - Read

Job - Build + Read

4] Create an API token for your git user.

Once you have assigned the permissions, log out from your current user and log in back using the git user which we have configured in earlier steps. Click you name (upper-right corner) → Click Configure (left-side menu) → Use Add new Token button to generate a new one then name it. Copy this token as we will require it in further steps.

3.4. Authorize Jenkins to connect with GitHub repositories

Navigate to the Jenkins dashboard and select the Credentials menu item. Select the System sub-menu” and the Global credentials domain. Click the Add credentials link. Select the Username with password credential type and enter your GitHub username and password in the corresponding fields. Set the ID field to github. Click OK to save the changes.

3.5. Add Plugins

Install the Google Kubernetes Engine plugin.

Navigate to the Manage Jenkins → Manage Plugins page. Select the Available tab and select the Google Kubernetes Engine plugin. Click the Install without restart button. Wait for the plugin to be installed and become available for use.

Install the Docker pipeline plugin.

Navigate to the Manage Jenkins → Manage Plugins page. Select the Available tab and select the Docker Pipeline plugin. Click the Install without restart button. Wait for the plugin to be installed and become available for use.

Install the Google Container Registry Auth plugin.

Navigate to the Manage Jenkins → Manage Plugins page. Select the Available tab and select the Google Container Registry Auth plugin. Click the Install without restart button. Wait for the plugin to be installed and become available for use.

4. Configure the GCP credentials

To allows Jenkins to deploy build artifacts on Google Container Registry and deploy applications on the GKE cluster, we will need to add GCP service account as credentials.

Creating a GCP service account.

gcloud iam service-accounts create $SA_NAME --display-name="$SA_NAME"

Attach the below roles to the service account to allow Jenkins to push/pull images to GCR and deploy the applications on GKE cluster.

We will assign following permissions to the GCP service account.

Storage Admin // To allow jobs to pull and push images on GCR.

Kubernetes Engine Developer // To allow jobs to deploy application GKE cluster

Use the following commands to assign permission on GCP service account.

gcloud projects add-iam-policy-binding $GCP_PROJECT_ID --member=serviceAccount:$SA_NAME@$GCP_PROJECT_ID.iam.gserviceaccount.com --role=roles/storage.admin gcloud projects add-iam-policy-binding $GCP_PROJECT_ID --member=serviceAccount:$SA_NAME@$GCP_PROJECT_ID.iam.gserviceaccount.com --role=roles/container.developer

Download the JSON-formatted key file.

gcloud iam service-accounts keys create $SA_NAME.json --iam-account $SA_NAME@$GCP_PROJECT_ID.iam.gserviceaccount.com

Navigate to the Jenkins dashboard and select the Credentials menu item. Select the System sub-menu” and the Global credentials domain. Click the Add credentials link. Select the Google Service Account from private key credential type and enter your GCP Project ID username and upload the GCP service account JSON key which we generated in previous step. Click OK to save the changes.

Navigate to the Jenkins dashboard and select the Credentials menu item. Select the System sub-menu” and the Global credentials domain. Click the Add credentials link. Select the Secret file credential type and upload the GCP service account JSON key which we generated in previous step. Click OK to save the changes.

5. Setup the Nginx Ingress Controller

5.1. Connect to application cluster

gcloud container clusters get-credentials application-cluster --zone $GCP_ZONE --project $GCP_PROJECT_ID

5.1. Deploy the Nginx Ingress Controller

Deploy the Nginx Ingress Controller for creating dynamic endpoints for our review application in the application cluster

kubectl apply -f review-apps-jenkins-gke/nginx-ingress/deployment.yaml

Note: For private clusters, you will need to either add an additional firewall rule that allows master nodes access to port 8443/tcp on worker nodes, or change the existing rule that allows access to ports 80/tcp , 443/tcp and 10254/tcp to also allow access to port 8443/tcp .

Verify all the nginx ingress controller components are running.

kubectl get all -n ingress-nginx

6. Configure review job

6.1. Create a job to deploy/destroy the review application

Click the New Item menu within Jenkins.

menu within Jenkins. Provide a name for your new item (e.g.review-job) and select Pipeline.

Configure the Job.

1] In Jenkins job, provide your GitHub repository URL when creating a new build job, we have the option of specifying the names of parameters that we expect to pass to the job in the POST that triggers the build. In this case, we would pass a single parameter payload.

2] Further down in the job configuration, we can specify that we would like to be able to trigger the build remotely (ie. that we want to allow github to trigger the build by posting to our URL with the payload)

Generate a 48 character strong token using below command.

head /dev/urandom | tr -dc A-Za-z0-9 | head -c 48 ; echo ''

Under Build Triggers →Trigger builds remotely, paste the token which is generate using above command.

master

3] Under Pipeline → Select SCM as Git. Provide your Repository URL and select your Credentials from the drop down. For Branch Specifier, keep it */* to make it applicable on all repositories. Finally, for the Script Path provide Jenkinsfile-review.

6.2. Configure the GitHub webhook

Now, as we have created the Jenkins with HTTPS endpoint we need to add it in our GitHub repository for reading the GitHub PR events.

1] Navigate to your GitHub repository and select Settings.

2] Select Add webhook under Webhooks.

3] Paste the Jenkins Webhook URL to the Payload URL.

where,

JENKINS_USER, JENKINS_USER_API_TOKEN :: The Jenkins user and API token we created in step 3.2. DNS_NAME :: The Jenkins enpoint domain name. JOB_NAME :: The name of the Jenkins job for review app JOB_TOKEN :: The 48 character token which we generated in step 4.1.

e.g.,

4] Select the content type as x-www-form-urlencoded.

5] Paste the WEBHOOK_VERIFY_TOKEN in the Secret property.

Provide the Authentication token which we generated in earlier steps while configuring the review-job.

6] And on the same page, select Let me select individual events and only tick mark the Pull requests from the checkbox list.

Once the GitHub Webhook is added you should receive a 200 response code for successful integration.

6.3. Update the environment variables in Jenkinsfile

Provide and update the environment variables in file Jenkinsfile-review for the review application

where,

GCP_PROJECT_ID :: The name of the GCP Project ID APP_NAME :: The name of the application. GCP_ZONE :: The zone of the GKE cluster GKE_CLUSTER_NAME :: The name of the GKE cluster

7. Configure staging job

7.1. Create a job to deploy the staging application

Click the New Item menu within Jenkins.

menu within Jenkins. Provide a name for your new item (e.g.staging-job) and select Pipeline.

Configure the Job.

1] In Jenkins job, provide your GitHub repository url when creating a new build job.

2] Under Build Triggers, enable the GitHub hook trigger for GITScm polling.

3] Under Pipeline → Select SCM as Git. Provide your Repository URL and select your Credentials from the drop down. For Branch Specifier, keep it */main to make it applicable on all repositories. Finally, for the Script Path provide Jenkinsfile-stage.

7.2. Configure the GitHub webhook

Now, as we have created the Jenkins with HTTPS endpoint we need to add it in our GitHub repository for reading the GitHub PR events.

1] Navigate to your GitHub repository and select Settings.

2] Select Add webhook under Webhooks.

3] Paste the Jenkins Webhook URL to the Payload URL.

where,

DNS_NAME :: The Jenkins enpoint domain name.

e.g.,

4] Select the content type as application/json.

6] And on the same page, select Just the push event.

Once the GitHub Webhook is added you should receive a 200 response code for successful integration.

7.3. Update the environment variables in Jenkinsfile

Provide and update the environment variables in file Jenkinsfile-stage for the staging application

where,

GCP_PROJECT_ID :: The name of the GCP Project ID APP_NAME :: The name of the application. GCP_ZONE :: The zone of the GKE cluster GKE_CLUSTER_NAME :: The name of the GKE cluster

7. Test the CICD pipeline

7.1. Deploy the application on staging

Make the following change at sample-app/index.html on the stage branch.

<body style="background-color:lightblue;"> <h1><p style="color:black">My v1</p></h1>

Commit the changes on the stage branch.

In the Jenkins the stage-job job will be triggered. Once the job is successful go ahead and access the application.

Access the stage application.

You can see the current version is v1 and background color is lightblue.

7.2. Create a pull request

Make the following change at sample-app/index.html on the stage branch.

<body style="background-color:lightgreen;"> <h1><p style="color:black">My v2</p></h1>

Commit the changes to a new patch branch.

On the stage open a new pull request from a patch branch to the stage branch.

In the Jenkins the review-job job will be triggered on the pull request open, re-opened and synchronize events.

Once the pull request is open, a new review application associated with the pull request number will be deployed in the GKE cluster.

Access the review application

The review application can be access using the nginx ingress endpoint on a dynamically generated route path.

You can see the current version is v2 and background color is lightgreen.

7.2. Approve the pull request

Once you approve the pull request there will be 2 workflow jobs triggered:

stage-job— Rollout the new changes on the staging environment.

review-job— Destroy the review application.

Access the review application

The ephemeral deployment will be destroyed.

Access the staging application.

Now, you can see the new changes are rolled out on the staging environment.

7.3. Reject the pull request

Once you reject the pull request the review-job pipeline will be triggered to destroy the review application.

Access the review application

The ephemeral deployment will be destroyed.

Questions?

If you have any questions, I’ll be happy to read them in the comments. Follow me on medium or LinkedIn.