How to ace the Google Cloud Associate Cloud Engineer Certification Exam!
By: Chris Herron
Tags: Gcp, Cloud Computing, Associate Cloud Engineer, Certification

1. Setting up a cloud solution environment

1.1 Setting up cloud projects and accounts

Creating projects

Select the project dropdown menu at the top of the Home menu, click New Project, enter a Project Name, then the Project ID is automatically assigned. A Location can be selected to give a parent organisation or folder.

On the command line type

gcloud projects create [Project ID] --folder [Folder ID] --labels [Labels] --name [Name] --organization [Organization]

To delete the project, go to the Home page for the project, then Project Settings, then Shut Down at the top (also in IAM & Admin, then Settings).

Creating and managing projects: https://cloud.google.com/resource-manager/docs/creating-managing-projects#console

Assigning users to predefined IAM roles within a project

Select IAM & Admin from the left menu, then IAM. Click Add up the top, then enter a member to add (a Gmail email, Google Group, service account, GSuite domain, or Cloud Identity domain) and select a role for them. You can select users in this window and click Remove up the top to delete.

On the command line type

gcloud [Project/Organization] add-iam-policy-binding [Project/Organization Name] --member user:user-email --role [Role ID]

Do the same but with remove-iam-policy-binding to delete a member from a role.

Granting, changing, and revoking access: https://cloud.google.com/iam/docs/granting-changing-revoking-access

Managing users in Cloud Identity (manually and automated)

Select IAM & Admin from the left menu, then Identity and Organisation. The first time you select this a window will appear saying to sign up to Cloud Identity to manage enterprise identities, and what to expect.

After this you’ll need to provide your name, email, country, and business name, amongst other information to verify ownership of the business domain.

Setting up Cloud Identity: https://cloud.google.com/identity/docs/setup

Enabling APIs within projects

In the left menu select APIs and Services, and then Library. Search for the required API, select it, and then click Enable in the page that appears. Some APIs like Compute Engine automatically enable when selecting them from the menu in the console.

They can be enabled from the command line by entering gcloud services enable SERVICE_NAME

Provisioning one or more Stackdriver workspaces

Note that the Stackdriver suite of applications is now referred to Cloud Monitoring/Cloud Logging, and is available from within the GCP console. In the left menu select Monitoring, then a workspace for the project will automatically be created, or add the project to an existing workspace. Stackdriver will then configure the workspace. You need to be a project owner, monitoring editor/admin, or Stackdriver Accounts Editor to create the workspace.

Creating Workspaces: https://cloud.google.com/monitoring/workspaces/create

1.2 Managing billing configuration

Creating one or more billing accounts

If no project is selected, click the left menu, then Billing. If a project is selected, still select Billing, but after that click the Billing Account dropdown at the top of the Overview window, then Manage Billing Accounts. In the window that appears, click Create Account. Give the new account a name and select a country for the currency. Then enter in payment details, then click Submit and enable billing.

Create, modify, or close your Cloud Billing Account: https://cloud.google.com/billing/docs/how-to/manage-billing-account

Linking projects to a billing account

If a billing account is already created, a billing account must be chosen when a project is created. If there is only one billing account, everything is automatically added to that. To switch billing accounts for an existing project, go to Billing, select the linked billing account, then from the left menu select Account Management. Select the project to change the billing account, then Change Billing

This can be done on the command line by

gcloud beta billing projects link [Project ID] --billing-account [Account ID]

Modify a project’s billing settings: https://cloud.google.com/billing/docs/how-to/modify-project

Establishing billing budgets and alerts

With the billing account administrator (or similar role), go to Billing, then Budgets & alerts, then Create Budget at the top of the window. Then define a name for the budget, which projects and products it applies to, how much the budget is, and what actions are taken after limits in the budget are reached. It’s also possible to connect a PubSub topic to the budget.

Set budgets and budget alerts: https://cloud.google.com/billing/docs/how-to/budgets

Setting up billing exports to estimate daily/monthly charges

You first need to make a BigQuery dataset to hold the billing exports. Select BigQuery from the navigation menu, select a project from the left, and then click Create Dataset on the right. Then go to Billing, and Billing export, then select the BigQuery dataset as the one to send the billing export to.

Set up Cloud Billing data export to BigQuery: https://cloud.google.com/billing/docs/how-to/export-data-bigquery-setup

1.3 Installing and configuring the command line interface (CLI), specifically the Cloud SDK

The Cloud SDK installation includes gcloud, gsutil, and bq. The normal installer is made for interactive installation, and a versioned archive of Cloud SDK should be used to automatically build the same version each time. For interactive installation download and run the installer, and follow the prompts. You’ll need a version of Python at 2.7.9 or higher, though you can install the needed Python version by default. You can then start the Cloud SDK shell, and run gcloud init to initialise the gcloud command, which includes authorising gcloud to use user account credentials, and sets properties like the current project, and default Compute Engine region and zone. These settings are local and persistent. The default installation doesn’t include App Engine extensions required to deploy an application using gcloud.

Using the Google Cloud SDK installer: https://cloud.google.com/sdk/docs/downloads-interactive

Initializing Cloud SDK: https://cloud.google.com/sdk/docs/initializing

2. Planning and configuring a cloud solution

2.1 Planning and estimating GCP product use using the Pricing Calculator

The pricing calculator is a form that enables you to select multiple GCP resources, and specify the details and configuration of those resources, to estimate their cost. This is only an estimate, and actual costs will vary.

https://cloud.google.com/products/calculator/

2.2 Planning and configuring compute resources

Selecting appropriate compute choices for a given workload (e.g. Compute Engine, Google Kubernetes Engine, App Engine, Cloud Run, Cloud Functions)

Compute Engine — Customisable VMs on demand, can add GPUs and TPUs — https://cloud.google.com/compute

Kubernetes Engine — Managed Kubernetes clusters for running containers, uses persistent disk for storage, can auto-scale — https://cloud.google.com/kubernetes-engine

App Engine — Platform as a service to run code, auto-scales — https://cloud.google.com/appengine

Cloud Run — Managed environment to run stateless containers, serverless and auto-scales, only pay when code is running, can be used with Anthos — https://cloud.google.com/run

Cloud Functions — Event based function as a service, run code when an event occurs — https://cloud.google.com/functions

Anthos — Managed environment to run containers in cloud or on-prem or edge devices, build, deploy and manage applications, consistent development and operations experience across all deployments — https://cloud.google.com/anthos

From Google’s ‘Preparing for the Google Cloud Associate Cloud Engineer Exam’ Coursera course: https://www.coursera.org/learn/preparing-cloud-associate-cloud-engineer-exam?authType=google#syllabus

Using preemptible VMs and custom machine types as appropriate

Preemptible VMs should be used when you have a fault-tolerant process or application and want to save money on compute costs, such as with batch processing. They cannot be automatically restarted, and automatically terminate after 24 hours. Custom machine types should be used to optimise processing/memory, or optimise costs.

Preemptible VM instances: https://cloud.google.com/compute/docs/instances/preemptible

Custom machine types: https://cloud.google.com/compute/docs/machine-types#custom_machine_types

2.3 Planning and configuring data storage options

Product choice (e.g. Cloud SQL, BigQuery, Cloud Spanner, Cloud. Bigtable)

Cloud SQL — Regional, fully managed mySQL/PostgreSQL/SQL Server. Manual scaling vertically and horizontally, handles transactional data — https://cloud.google.com/sql

Cloud Spanner — Regional, multi-regional, or global. First horizontally scalable relational database service. For transactional data — https://cloud.google.com/spanner

BigQuery — Multi-regional, serverless, column-store data warehouse that we can run queries on and use for analytics. Scales internally — https://cloud.google.com/bigquery

Cloud Bigtable — Low-latency, high-throughput NoSQL database, wide-column store. For large analytical and operational workloads — https://cloud.google.com/bigtable

Choosing storage options (e.g. regional, multi-regional, nearline, coldline)

Location options:

Regional — Store data in a specific place. Optimise latency and network bandwidth for data consumers

Dual-region — Store data in a specific pair of places. Similar advantages to regions, but higher availability

Multi-regional — Stored in a large geographic area with two or more specific places. Use to serve content to consumers outside the Google network

Objects stored in dual-region or multi-region are geo-redundant.

Storage classes:

Standard — Best for short-term storage and frequently accessed data

Nearline — Best for backups and data accessed less than once a month

Coldline — Best for disaster recovery and data accessed less than once a quarter

Archive — Best for long-term digital preservation of data accessed less than once a year

All storage classes are accessible worldwide, low latency, high durability.

Storage classes: https://cloud.google.com/storage/docs/storage-classes

Persistent Disks — High performance block storage for virtual machines and containers

Local SSD — Ephemeral locally-attached block storage for virtual machines and containers

Compute engine storage options: https://cloud.google.com/compute/docs/disks

Cloud Filestore — Fully managed, scalable file storage with predictable performance — https://cloud.google.com/filestore

Cloud Storage for Firebase — Scalable storage for user-generated content from your app — https://firebase.google.com/docs/storage

Cloud Memorystore — Fully managed in-memory data store service for Redis built on scalable, more secure, highly available infrastructure. Ideal for building application caches and sub-millisecond data access using Redis — https://cloud.google.com/memorystore

Cloud Firestore (previously Datastore) — Fast, fully managed, serverless, cloud-native NoSQL document database. Ideal for building client side mobile and web applications, gaming leaderboards, and user presence at global scale — https://cloud.google.com/firestore

Firebase Realtime Database — Cloud-hosted NoSQL database that lets you store and sync data between your users in real time. Ideal for creating onboarding flows, rolling out new features, and building serverless apps — https://firebase.google.com/docs/database

From Google’s ‘Preparing for the Google Cloud Associate Cloud Engineer Exam’ Coursera course: https://www.coursera.org/learn/preparing-cloud-associate-cloud-engineer-exam?authType=google#syllabus

From Google’s ‘Preparing for the Google Cloud Associate Cloud Engineer Exam’ Coursera course: https://www.coursera.org/learn/preparing-cloud-associate-cloud-engineer-exam?authType=google#syllabus

From Google’s ‘Preparing for the Google Cloud Associate Cloud Engineer Exam’ Coursera course: https://www.coursera.org/learn/preparing-cloud-associate-cloud-engineer-exam?authType=google#syllabus

2.4 Planning and configuring network resources

Differentiating load balancing options

The right option depends on if internal or external traffic needs to be balanced, and the type of traffic, e.g. HTTP(S), TCP, UDP. Use global load balancing when backends are distributed across multiple regions and you want to provide access using a single anycast IP address, that may require IPv6. Use regional load balancing when the backends are in one region, and you only require IPv4 termination.

Choosing a load balancer — https://cloud.google.com/load-balancing/docs/choosing-load-balancer

For Kubernetes services:

ClusterIP — Internal clients send requests to a stable internal IP address

NodePort — Clients send requests to the IP address of a node on one or more nodePort values that are specified by the service (extension of ClusterIP)

LoadBalancer — Clients send requests to the IP address of a network load balancer (extension of NodePort)

ExternalName — Internal clients use the DNS name of a service as an alias for an external DNS name

Headless — Can be used where you want a pod grouping but don’t need a stable IP address

Kubernetes services — https://cloud.google.com/kubernetes-engine/docs/concepts/service

Identifying resource locations in a network for availability

Spread resources across multiple availability zones and load balance across them to prevent single points of failure.

Configuring Cloud DNS

Go to Network Services then Cloud DNS, the first time you access this the Cloud DNS API will be enabled. You need to have a domain name before setting this up (can get this through Google Domains) and a valid IP address. Then you need to create a new DNS zone, giving it a name, the domain suffix, and set DNSSEC off. Next select the created zone, and click Add Record Set. Choose the type of address, then enter the IP address. Next repeat this, but for a CNAME record instead of an A record, and enter the DNS name as www. The canonical name is the domain name. After this, you need to update the name servers in Google Domains.

Setting up a domain using Cloud DNS: https://cloud.google.com/dns/docs/tutorials/create-domain-tutorial

3. Deploying and implementing a cloud solution

3.1 Deploying and implementing Compute Engine resources

Launching a compute instance using Cloud Console and Cloud SDK (glcoud) (e.g. assign disks, availability policy, SSH keys)

Go to Compute Engine, then VM Instances. Click on Create to make an instance. There are options to create an instance from a template, from a machine image, or deploy a solution from the marketplace. If you’re not using one of these options, then enter in labels, region, and zone, then choose the machine configuration, e.g. general purpose, memory optimised, or compute optimised, then select the series, and machine type (can be custom). You can also add GPUs, a container, choose the boot disk, set up Identity and API access, firewalls, provide a startup script and metadata, make it preemptible, set up encryption on the disk, and manage networking tags. Once everything required is complete, click Create.

In gcloud the command to create an instance is similar to

gcloud compute instances create VM_NAME

[--image IMAGE | --image-family IMAGE_FAMILY]

--image-project IMAGE_PROJECT

--subnet [SUBNET_NAME]

--zone [ZONE_NAME]

--machine-type=n1-standard-1

where you can provide flags to specify the particular configuration you require.

Creating an autoscaled managed instance group using an instance template

To create an instance template go to Compute Engine, then Instance Templates. Click Create Instance Template, then follow similar steps as for creating an instance, but without entering in a region or zone.

For the managed instance group, go to Compute Engine then Instance Groups, and Create Instance Group. Choose New Managed Instance Group, then select whether the group should be in one zone or multiple, then the region/zone, choose the instance template, and set the Auto-scaling mode to Auto-scale. Auto-scaling metrics can be set, as well as the min and max number of instances.

In gcloud use (adding in the desired configuration settings as flags)

gcloud compute instance-templates create INSTANCE_TEMPLATE_NAME gcloud compute instance-groups managed create instance-group-name \

--base-instance-name base-name \

--size size \

--template instance-template \

--zone zone

Creating instance templates: https://cloud.google.com/compute/docs/instance-templates/create-instance-templates

Creating managed instance groups: https://cloud.google.com/compute/docs/instance-groups/creating-groups-of-managed-instances

Generating/uploading a custom SSH key for instances

Use the OS Login API to avoid the risk of manual key management. This allows you to use Compute Engine IAM roles to manage SSH access to instances. Otherwise you would need to create an SSH key to let users access a Linux instance through third-party tools. You can create a new key using ssh-keygen and specify the filename to use for the generated keys, and the username of the person they apply to. Go to Compute Instances then Metadata, and in the SSH Keys tab, click Add SSH keys, and copy the public key data in and save. This option is riskier, and only recommended if OS Login in unavailable.

Setting up OS Login: https://cloud.google.com/compute/docs/instances/managing-instance-access

Managing SSH keys in metadata: https://cloud.google.com/compute/docs/instances/adding-removing-ssh-keys

Configuring a VM for Stackdriver monitoring and logging

You need to have a supported VM instance with a supported OS first, and 250 MB of available memory for the monitoring agent. The instance also needs credentials to be able to communicate with Cloud Monitoring/Cloud Logging. Open a terminal connection to the instance using SSH, make sure you have sudo access, change to a directory where you have write access, and download the Cloud Monitoring/Cloud Logging agent’s package repository. Then install the agent, and start it. It is best practice to run the Logging agent on all VMs. The Logging agent requires a minimum 250 MiB of memory, but 1 GiB is recommended. The same steps are followed for the Monitoring/Logging agents, but a different URL is used to download their agents.

Installing the Cloud Monitoring agent on a single VM: https://cloud.google.com/monitoring/agent/installation

Installing the Cloud Logging agent on a single VM: https://cloud.google.com/logging/docs/agent/installation

Assessing compute quotas and requesting increases

Go to IAM and Admin then Quotas to see all Quotas, then filter to those related to Compute Engine. Use this page to request quota increases as well by looking at the details for the quota, then clicking Edit Quotas. You’ll need to provide your contact details as well as what you’d like to change the quota to.

You can also enter this on the command line to view quotas

gcloud compute project-info describe --project myproject

VM quotas are regional, networking and load balancing quotas are global.

Working with quotas: https://cloud.google.com/docs/quota

Checking your quota: https://cloud.google.com/compute/quotas#checking_your_quota

Installing the Stackdriver Agent for monitoring and logging

See above.

3.2 Deploying and implementing Google Kubernetes Engine resources

Deploying a Google Kubernetes Engine cluster

The Kubernetes Engine API needs to be enabled before a cluster can be created. Go to Kubernetes Engine then Clusters, and click Create Cluster. Enter cluster basics, such as name, and whether it’s zonal/regional, where it is, and the Kubernetes version. You can add as many node pools as required. Each node has a name, its version of Kubernetes, a number of nodes, an option to enable auto-scaling with the min/max number of nodes, enable auto-upgrade and auto-repair, and enable surge-upgrade. For the nodes select the OS, machine type for CPU/memory, persistent disk, local SSD, if the nodes can be preemptible, the maximum number of pods per node, and network tags. In Security specify the service account and scopes. In Metadata labels can be added. For cluster properties it’s possible to set auto-scaling and auto-provisioning, whether the cluster is public or private, the node subnet, load balancing, and features like Cloud TPU, Istio, and Anthos can be turned on.

In gcloud a single-zone cluster can be made using

gcloud container clusters create cluster-name --zone compute-zone

A multi-zone cluster can be made using

gcloud container clusters create cluster-name --zone compute-zone --node-locations compute-zone,compute-zone,[...]

A regional cluster can be made using

gcloud container clusters create cluster-name --region compute-region [--node-locations compute-zone,compute-zone...]]

After these, kubectl needs to be configured from the command line so that the cluster can be accessed:

gcloud container clusters get-credentials cluster-name

Creating a single-zone cluster: https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-cluster#using-gcloud-init

Creating a multi-zone cluster: https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-multi-zonal-cluster

Creating a regional cluster: https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-regional-cluster

Configuring cluster access for kubectl: https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl#generate_kubeconfig_entry

Deploying a container application to Google Kubernetes Engine using pods

This can be done in the Cloud Shell in the console, and in a local command line. You first need to create a Docker image using an application and a Dockerfile. Then upload the image to a registry so that GKE can download and run it. This can be done with Google Container Registry or Docker. Create a container cluster, then deploy the application by first creating a deployment and all resources with

kubectl apply -f [DEPLOYMENT_FILE]

(This is preferable since it uses a YAML manifest file that specifies the entire configuration). If you’re not using a YAML file, then use

kubectl create deployment [NAME] [IMAGE]

and then you can expose the application to the internet with

kubectl expose deployment [NAME] [LOAD BALANCER TYPE] [PORT]

which will create the Service resource automatically. An update can be applied using

kubectl set image DEPLOYMENT

Deploying a stateless application: https://cloud.google.com/kubernetes-engine/docs/how-to/stateless-apps

Configuring Google Kubernetes Engine application monitoring and logging

GKE has native integration with Cloud Monitoring and Cloud Logging. Kubernetes Engine Monitoring is enabled by default and manages Monitoring and Logging Services together, showing cluster metrics, pods/containers/services etc, and metrics/logs for pods and containers. To set up Kubernetes Engine Monitoring you need to be an Owner of the project, and the project needs to be associated with a Cloud Monitoring/Logging workspace.

Installing Cloud Operations for GKE support: https://cloud.google.com/stackdriver/docs/solutions/gke/installing

3.3 Deploying and implementing App Engine, Cloud Run, and Cloud Functions resources

Deploying an application, updating scaling configuration, versions, and traffic splitting

An app can be deployed from the command line with

gcloud app deploy app.yaml index.yaml

The project and version can be specified through —-project and -v flags. Cloud Build builds a container image of the application to run in App Engine standard. New versions are deployed with

gcloud app deploy app.yaml

where the YAML file has been updated for this version. A new version number is made automatically, and traffic is routed to the new version. Other configuration files can be deployed separately by using

gcloud app deploy cron.yaml\dispatch.yaml\index.yaml

Multiple services can also be updated at once through

gcloud app deploy service1/app.yaml service2/app.yaml

Built container images are stored in the app-engine folder in Container Registry. You can view the application with

gcloud app browse

To test before deployment, add a --no-promote flag when deploying a new version, then perform the tests. You can switch to the new version in the Cloud Console by clicking that version and then Migrate Traffic.

For traffic splitting, select the versions you want to split traffic between in the console, then click Split Traffic, the method to use to split traffic, and the percentage of traffic each version should receive. This can also be done through the command line:

gcloud app services set-traffic [MY_SERVICE] --splits [MY_VERSION1]=[VERSION1_WEIGHT],[MY_VERSION2]=[VERSION2_WEIGHT]

--split-by [IP_OR_COOKIE]

Scaling is configured through the app.yaml files that are deployed, which can use an automatic_scaling element with min/max number of instances, cpu utilisation, latency, etc. There’s also basic_scaling and manual_scaling. The scaling supported depends on the instance type.

Testing and deploying your application: https://cloud.google.com/appengine/docs/flexible/python/testing-and-deploying-your-app#ensuring_successful_deployment

Splitting traffic: https://cloud.google.com/appengine/docs/flexible/python/splitting-traffic

For Cloud Run, the Cloud Run API first has to be enabled. An app can be deployed in the console by going to Cloud Run, clicking Create Service, and then filling in the form to specify the deployment platform and service name. A container image must be in Container Registry or Artifact Registry, and can be selected during this process.

This can be done in gcloud using

gcloud run deploy SERVICE --image gcr.io/PROJECT-ID/IMAGE

Deploying container images: https://cloud.google.com/run/docs/deploying#yaml

Deploying an application that receives Google Cloud events (e.g. Cloud Pub/Sub events, Cloud Storage object notification events)

A Cloud Function will run the given code when it’s triggered by some event. To create a Cloud Function go to Cloud Functions, specify the allocated amount of memory, what triggers the code, what the event type is, and then provide the code to run.

This can also be achieved in gcloud using

gcloud functions deploy NAME --entry-point NAME --runtime RUNTIME TRIGGER [FLAGS...]

Deploying from Cloud Console: https://cloud.google.com/functions/docs/deploying/console

Deploying from Your Local Machine: https://cloud.google.com/functions/docs/deploying/filesystem

3.4 Deploying and implementing data solutions

Initializing data systems with products (e.g. Cloud SQL, Cloud Datastore, Big Query, Cloud Spanner, Cloud Pub/Sub, Cloud Bigtable, Cloud Dataproc, Cloud Dataflow, Cloud Storage)

Cloud Storage — Go to Storage then Browser, then click on Create Bucket to make a bucket. Choose where the bucket is located, the storage class, and how access is handled. Data can be uploaded by dragging into a bucket, using the Upload Files or Upload Folder buttons, or through the command line.

In gcloud a bucket can be created using

gsutil mb -p PROJECT_ID -c STORAGE_CLASS -l BUCKET_LOCATION -b on gs://BUCKET_NAME

Objects can be copied into the bucket using

gsutil cp OBJECT_LOCATION gs://DESTINATION_BUCKET_NAME/

Creating storage buckets: https://cloud.google.com/storage/docs/creating-buckets#storage-create-bucket-gsutil

Uploading objects: https://cloud.google.com/storage/docs/uploading-objects

BigQuery — Go to BigQuery, then click on Create Dataset. You can also click on Add Data then Create Connection to retrieve data that way. Data can be loaded from Cloud Storage, your machine, streaming inserts, or Dataflow. If loading from Cloud Storage, the data can be loaded from any storage class, and the bucket should be in the same location as the BigQuery dataset. The filename needs to be provided, in the form gs://mybucket/myfile.csv . Data from a machine can be loaded using the Cloud Console, bq load command, API, or client libraries. BigQuery expects data to be in CSV and UTF-8 format for these two methods. If streaming inserts are being used, then data can be streamed into existing tables using the tabledata.insertAll API method.

When loading data in the console, click Create Table for the dataset, then in the page that appears, choose to create the table from Cloud Storage, Upload (direct from machine), make an empty table, use Google Drive, or use Cloud Bigtable. We can make the tables manually ourselves, or use schema auto-detection.

The gcloud command to create a dataset is

bq --location=location mk \

--dataset \

--default_table_expiration integer1 \

--default_partition_expiration integer2 \

--description description \

project_id:dataset

and to create a table

bq mk \

--table \

--expiration integer \

--description description \

--label key:value, key:value \

project_id:dataset.table \

schema

Creating datasets: https://cloud.google.com/bigquery/docs/datasets

Creating and using tables: https://cloud.google.com/bigquery/docs/tables

Cloud Bigtable — NoSQL database. Go to Bigtable, then click on Create Instance. Choose the storage type (SSD/HDD), the region/zone, the number of nodes in the cluster, and any replicated clusters if required. After that click Create. Data can be imported to the database using Dataflow.

An instance with a single cluster can be created in gcloud using

gcloud bigtable instances create INSTANCE_ID \

--cluster=CLUSTER_ID \

--cluster-zone=CLUSTER_ZONE \

--display-name=DISPLAY_NAME \

[--cluster-num-nodes=CLUSTER_NUM_NODES] \

[--cluster-storage-type=CLUSTER_STORAGE_TYPE] \

Creating a Cloud Bigtable instance: https://cloud.google.com/bigtable/docs/creating-instance#gcloud

Cloud SQL — Go to SQL, then Create Instance, or Migrate Data to migrate an existing SQL database. If creating an instance you’ll then need to choose the database engine, either MySQL/PostgreSQL/SQL Server, then set the instance ID, password for root user, location, and database version. Connectivity, machine type, backups, maintenance and labels can also be set.

In gcloud, an instance can be created using

gcloud sql instances create [INSTANCE_NAME] --tier=[MACHINE_TYPE] --region=[REGION]

Add in the —-database-version flag to choose the SQL version of the instance.

Creating instances: https://cloud.google.com/sql/docs/mysql/create-instance

Cloud Spanner — SQL database. The Cloud Spanner API needs to be enabled first. Like Bigtable, you then need to create an instance and load data through the API. Provide a name and ID, set whether it’s regional or multi-regional, and set the number of nodes before creating the instance.

In gcloud use

gcloud spanner instances create [INSTANCE-ID] --config=[INSTANCE-CONFIG] \

--description="[INSTANCE-NAME]" --nodes=[NODE-COUNT]

Creating and managing instances: https://cloud.google.com/spanner/docs/create-manage-instances

Cloud Firestore — NoSQL database for web/mobile applications. Click on Firestore, then you will need to select a Cloud Firestore mode, either Native mode or Datastore mode (Firestore is the next generation of Datastore) which will be permanent for the project. Native mode enables all of Firestore’s features, with offline support and real-time synchronisation, and can scale to millions of concurrent clients and focuses on real-time and offline data persistence. Datastore mode uses Datastore’s system on top of Firestore’s storage layer, and scales to millions of writes per second and has no limit on the number of writes. You then need to choose a location, which is also permanent, and applies to the project’s App Engine too. After this the database can be created.

Data can be brought into the Firestore database using the API/client libraries.

About Firestore: https://cloud.google.com/firestore

Cloud Pub/Sub — Click on Pub/Sub, then Create Topic. Enter an ID and set the encryption type, then click Create Topic. After a topic is created, permissions, labels, and storage policy can be set. In the Subscriptions menu click Create Subscription, provide an ID, select a Pub/Sub topic, set the delivery type to pull/push, set the expiry details for the subscription, acknowledgment deadline, message retention duration, dead lettering, then Create. Snapshots can be created for the message acknowledgement state of a subscription at a point in time.

In gcloud a topic and subscription can be created using

gcloud pubsub topics create my-topic

gcloud pubsub subscriptions create my-sub --topic=my-topic

Quickstart using the gcloud command-line tool: https://cloud.google.com/pubsub/docs/quickstart-cli

Cloud Dataproc — The Cloud Dataproc API needs to be enabled first. Click on Dataproc, then Create Cluster. Provide a name, location, cluster mode (set up of master and worker nodes), set the machine processor type for the master node, disk size and type, worker node machine type and disk size and type, number of nodes and local SSDs, the auto-scaling policy, if any nodes should be preemptible, the network it’s on, cloud storage staging bucket, Dataproc OS image, initialisation actions, cluster properties, metadata, security, labels, and encryption.

In gcloud use

gcloud dataproc clusters create cluster-name \

--region=region

It’s also possible to provide a YAML file in this command to define how the cluster is configured.

Create a cluster: https://cloud.google.com/dataproc/docs/guides/create-cluster

Cloud Dataflow — Click on Dataflow, then Create Job From Template or Create Job From SQL. Specify a name, regional endpoint, and select a template, or write the SQL in the BigQuery query editor then select Create Cloud Dataflow Job.

The data pipeline can also be written using the Apache Beam SDK in the coding language of your choice, and then the pipeline is run using the Dataflow runner.

Quickstart using Python: https://cloud.google.com/dataflow/docs/quickstarts/quickstart-python

From Google’s ‘Preparing for the Google Cloud Associate Cloud Engineer Exam’ Coursera course: https://www.coursera.org/learn/preparing-cloud-associate-cloud-engineer-exam?authType=google#syllabus

Loading data (e.g. command line upload, API transfer, import/export, load data from Cloud Storage, streaming data to Cloud Pub/Sub)

Loading data into Storage and BigQuery is covered above.

The command line upload to Cloud Storage is

gsutil cp [OBJECT_LOCATION] gs://[DESTINATION_BUCKET_NAME]/

The command line upload to BigQuery from a local machine or Cloud Storage is

bq --location=LOCATION load \

--source_format=FORMAT \

PROJECT_ID:DATASET.TABLE \

PATH_TO_SOURCE \

SCHEMA

Loading data from a local data source: https://cloud.google.com/bigquery/docs/loading-data-local#bq

For Bigtable, you can use Dataflow to load data from Cloud Storage. There are also data APIs and client libraries that allow you to write data to tables.

Importing and exporting data: https://cloud.google.com/bigtable/docs/import-export

For Cloud SQL, an SQL dump file can be imported through the Console or the command line, provided the file is in a Storage bucket. To do this go to the instance in the Console, click Import, then select the file. The command line version is

gcloud sql import sql [INSTANCE_NAME] gs://[BUCKET_NAME]/[IMPORT_FILE_NAME] \

--database=[DATABASE_NAME]

All SQL servers can do this. PostgreSQL and mySQL can also import CSV files.

Importing data into Cloud SQL: https://cloud.google.com/sql/docs/mysql/import-export/importing

For Cloud Spanner, data can be inserted, edited, or deleted through the console by going to the instance, then the database, then the table, then finding the row or inserting a row in the data tab. This can also be done using Data Manipulation Language (DML) statements either in the Console on the database page, or using gcloud similar to

gcloud spanner databases execute-sql example-db --instance=test-instance \

--sql="INSERT Singers (SingerId, FirstName, LastName) VALUES (1, 'Marc', 'Richards')"

or using a client library in another code language. Mutations in a client language can also be used, or gcloud can be used through the gcloud spanner rows command to insert, update, or delete.

gcloud spanner rows insert --instance=INSTANCE_ID --database=DATABASE_ID \

--table=TABLE_NAME \

--data=COL_NAME_1=COL_VALUE_1,COL_NAME_2=COL_VALUE_2,

COL_NAME_3=COL_VALUE_3,\

...,COL_NAME_N=COL_VALUE_N

Data can be imported from Cloud Spanner AVRO files that are in Storage, from non-Cloud Spanner databases by using AVRO files in Storage and a JSON manifest file with Dataflow, or similarly with CSV.

Inserting, updating, and deleting data using the gcloud command-line tool: https://cloud.google.com/spanner/docs/modify-gcloud

Importing Cloud Spanner Avro files: https://cloud.google.com/spanner/docs/import

Importing and exporting data in CSV format: https://cloud.google.com/spanner/docs/import-export-csv

For Cloud Firestore, the client library can be used to create, update, or delete documents from the database. Transactions/batched writes can also be conducted on up to 500 documents. This is done through the client library with .batch() and .commit() commands. Everything in the batch has to succeed or nothing is committed. Cloud Firestore exports can be loaded into Firestore again or into BigQuery. To import data the document needs to be in Storage, then use a gcloud command similar to the following or a client library to import the data:

gcloud firestore import gs://[BUCKET_NAME]/[EXPORT_PREFIX]/

Exporting and importing data: https://cloud.google.com/firestore/docs/manage-data/export-import

For Cloud Pub/Sub, you need to use a client library and write code that is designed to publish messages to a topic. After authorisation, publishing can be done via the command line as

gcloud pubsub topics publish my-topic --message "hello"

or through the client library for a coding language. If low latency is required, turn batching off.

Publishing messages to topics: https://cloud.google.com/pubsub/docs/publisher#gcloud

3.5 Deploying and implementing networking resources

Creating a VPC with subnets (e.g. custom-mode VPC, share VPC)

Go to VPC Network, then Create VPC Network. Choose custom to manually select where each subnet is, or automatic to have a subnet in each region. Next specify a private IP address range, dynamic routing mode, and DNS server policy. For an automatic VPC you can also select the firewall rules that will apply at this stage.

Shared VPC is only available for projects within an organisation. It allows you to share subnets in a host project with other service projects in the same organisation. This is referred to as XPN in the API and command line. A Shared VPC Admin needs to go to the Shared VPC page, then enable a project to act as the Shared VPC host project. Then click on Set Up Shared VPC. It’s possible to share all subnets if all subnets should be accessed by service projects, or individual subnets if each service project should have its own subnet. Then go to the Attached Projects tab, click Attach Project, select the projects to attach, and then either share all subnets or use individual subnets for each project.

Using VPC networks: https://cloud.google.com/vpc/docs/using-vpc

Provisioning Shared VPC: https://cloud.google.com/vpc/docs/provisioning-shared-vpc

Launching a Compute Engine instance with custom network configuration (e.g. internal-only address, Google private access, static external and private IP address, network tags)

You can provide network tags and choose the VPC/subnet that a compute engine instance belongs to when the instance is being created. Google Private access is set when creating the VPC. The VPC and instance need to be available in the same region. During set up you can also set a static external IP address, which can be reserved in the VPC Network — External IP Addresses page.

Configuring Private Google Access: https://cloud.google.com/vpc/docs/configure-private-google-access

Reserving a static external IP address: https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address

Creating ingress and egress firewall rules for a VPC (e.g. IP subnets, tags, service accounts)

Go to VPC Networks — Firewall Rules, or look at the VPC details then click on the Firewall Rules tab. Click Add Firewall Rule, specify the priority of the rule, direction of traffic, action on match, whether it applies to everything in the network/given network tags/given service accounts, source/destination filter, and the protocols/ports it applies to.

Firewall rules can also be created in gcloud using

gcloud compute firewall-rules create NAME \

[--network NETWORK; default="default"] \

[--priority PRIORITY;default=1000] \

[--direction (ingress|egress|in|out); default="ingress"] \

[--action (deny | allow )] \

[--target-tags TAG,TAG,...] \

[--target-service-accounts=IAM Service Account,IAM Service Account,...] \

[--source-ranges CIDR-RANGE,CIDR-RANGE...] \

[--source-tags TAG,TAG,...] \

[--source-service-accounts=IAM Service Account,IAM Service Account,...] \

[--destination-ranges CIDR-RANGE,CIDR-RANGE...] \

[--rules (PROTOCOL[:PORT[-PORT]],[PROTOCOL[:PORT[-PORT]],...]] | all ) \

[--disabled | --no-disabled] \

[--enable-logging | --no-enable-logging] \

[--logging-metadata logging_metadata]

Using firewall rules: https://cloud.google.com/vpc/docs/using-firewalls

Creating a VPN between a Google VPC and an external network using Cloud VPN

Use Cloud VPN to securely connect on-premises machines to a VPC. Go to Hybrid Connectivity, then VPN, then click on Create VPN. You can choose between high availability VPN or classic. After this specify the network, region, and IP address for the VPN gateway, and VPN tunnel details.

Creating an HA VPN gateway to a peer VPN gateway: https://cloud.google.com/network-connectivity/docs/vpn/how-to/creating-ha-vpn

Creating a load balancer to distribute application network traffic to an application (e.g. Global HTTP(S) load balancer, Global SSL Proxy load balancer, Global TCP Proxy load balancer, regional network load balancer, regional internal load balancer)

Go to Network Services, then Load Balancing, and click on Create Load Balancer. Choose between HTTP(S), TCP, or UDP load balancers. They can all be internet facing or internal, and single region, but only HTTP(S) and TCP can be multi-region. Then select the backend configuration, host/path rules, and the frontend configuration, then create. TCP/SSL doesn’t need the host/path rules, but does need the frontend and backend configuration, and same for UDP.

Cloud Load Balancing Overview: https://cloud.google.com/load-balancing/docs/load-balancing-overview

Setting up Internal HTTP(S) Load Balancing: https://cloud.google.com/load-balancing/docs/l7-internal/setting-up-l7-internal

3.6 Deploying a solution using Cloud Marketplace

Browsing Cloud Marketplace catalog and viewing solution details

The Marketplace can be accessed directly from the Console menu. Just search and filter for what’s required then select a product for more information.

Deploying a Cloud Marketplace solution

Some solutions have click to deploy, and will launch quickly once selected. Others will provide instructions.

3.7 Deploying application infrastructure using Cloud Deployment Manager

Developing Deployment Manager templates

Go to Deployment Manager, then Deployments, and enable the Deployment Manager API. This shows anything that has been deployed from the Marketplace, and also user-created deployments, which need to be specified as a YAML template. It’s good to have templates for lots of little things so that they can be reused, and have them be as generalised or specific as required. They can be written in Jinja 2.8 or Python 3.x. A configuration written in YAML will import the provided templates during deployment, which are expanded with parameters that are specified in the configuration file.

A Python template needs a generate_config(context) method within it, and must return a Python dictionary. Properties can be created in the template using

context.properties["PROPERTY_NAME"]

These properties are then specified in the configuration YAML file to create the required resource.

When the template is written it is imported into the configuration YAML using

imports:

- path: path/to/my_vm_template.jinja

name: my_renamed_template.jinja

Imported templates are then used as resources within the YAML file

imports:

- path: vm-template.py resources:

- name: my-vm

type: vm-template.py

Templates can also be deployed directly from the command line using gcloud, similar to

gcloud deployment-manager deployments create my-igm

--template vm_template.jinja

--properties zone:us-central1-a,machineType:n1-standard-1,image:debian-9

Understanding reusable templates: https://cloud.google.com/deployment-manager/docs/step-by-step-guide/create-a-template

Creating a Basic Template: https://cloud.google.com/deployment-manager/docs/configuration/templates/create-basic-template

Launching a Deployment Manager template

A deployment configuration should be specified in a YAML file, and then the deployment can be launched from the command line using gcloud, similar to:

gcloud deployment-manager deployments create example-cluster\

--config cluster-1/cluster.yaml

After a configuration file is complete with included templates, it’s possible to preview the configuration on the command line using

gcloud deployment-manager deployments create example-config\

--config configuration-file.yaml --preview

Creating a Deployment using gcloud or the API: https://cloud.google.com/deployment-manager/docs/deployments

4. Ensuring a successful operation of a cloud solution

4.1 Managing Compute Engine resources

Managing a single VM instance (e.g. start, stop, edit configuration, or delete an instance)

In Compute Engine — VM Instances first select an instance to see the options to start, stop, or delete it. The details for the instance will become visible when it’s selected, and you can select Edit to change its configuration and save it.

Useful gcloud commands are

gcloud compute instances create my-instance

gcloud compute instances delete my-instance --zone us-central1-a

gcloud compute instances reset example-instance --zone us-central1-a

gcloud compute instances start example-instance --zone=us-central1-a

gcloud compute instances stop test-instance

gcloud compute instances update example-instance --zone=us-central1-a --update-labels=k0=value1,k1=value2 --remove-labels=k3

Stopping and starting an instance: https://cloud.google.com/compute/docs/instances/stop-start-instance

Updating instance properties: https://cloud.google.com/compute/docs/instances/update-instance-properties

SSH/RDP to the instance

You can SSH to an instance in the console by selecting the instance and clicking on SSH. Using gcloud you can use

gcloud compute ssh my-instance --zone us-central1-a

You can also use scp to copy files from an instance in gcloud using

gcloud compute scp ~/file-1 my-instance:~/remote-destination --zone us-central1-a

Connecting to instances: https://cloud.google.com/compute/docs/instances/connecting-to-instance

Attaching a GPU to a new instance and installing CUDA libraries

When creating an instance, expand CPU platform and GPU, then click Add GPU, then choose the type and number of GPUs. Note that GPUs are only available in certain regions/zones, and only for N1 series processors. For the CUDA libraries you need a NVIDIA account, and then you need to download the repository onto the VM, then install and update the package.

GPUs on Compute Engine: https://cloud.google.com/compute/docs/gpus

Adding or removing GPUs: https://cloud.google.com/compute/docs/gpus/add-gpus

Installing GPU drivers: https://cloud.google.com/compute/docs/gpus/install-drivers-gpu

Viewing current running VM inventory (instance IDs, details)

Go to Compute Engine — VM Instances to see currently running VMs, and select them to see more details.

In gcloud use

gcloud compute instances list

to get information on all instances, and

gcloud compute instances describe my-instance --zone us-central1-a

to get information on one instance.

Working with snapshots (e.g. create a snapshot from a VM, view snapshots, delete a snapshot)

Snapshots are taken of the persistent disks attached to VMs to back them up and restore them. They are incremental and automatically compressed, so take less time and are cheaper than full images. The first successful snapshot is a full snapshot, and then all subsequent snapshots refer to the most recent successful snapshot. Snapshots can be seen in the console by going to Compute Engine then Snapshots, or to see all snapshots in gcloud use

gcloud compute snapshots list

and to see the details for one snapshot

gcloud compute snapshots describe [SNAPSHOT_NAME]

When creating a snapshot you specify a storage location, which can either be a regional or multi-regional bucket. The snapshot can be used anywhere regardless of where it’s stored. If a location isn’t specified it’s stored in the closest multi-region. To prepare to take the snapshot, it’s ideal to flush the disk’s buffers, then freeze or unmount the filesystem, take the snapshot, then remount the disk. Best practice is one snapshot per hour.

To create a snapshot in the console, go to Compute Engine — Snapshots, then click Create a Snapshot or Create Snapshot Schedule. Choose the source disk and select a snapshot storage location.

In gcloud the command is

gcloud compute disks snapshot [DISK_NAME] \ --storage-location [STORAGE_LOCATION]

or to have a custom storage location

gcloud compute disks snapshot [DISK_NAME] --region [REGION] \ --storage-location [STORAGE_LOCATION]

To restore a snapshot go to Compute Engine — Disks, then Create a New Disk, choose a type and location, and under Source Type click Snapshot, and then choose the snapshot to restore.

The gcloud is

gcloud compute disks create [DISK_NAME] --size [DISK_SIZE] --source-snapshot [SNAPSHOT_NAME] --type [DISK_TYPE]

If a snapshot is deleted, it isn’t completely removed, since some of its data may be moved into a subsequent dependent snapshot. To delete a snapshot, go to Compute Engine — Snapshots, select the snapshots to delete, then click delete. In gcloud the command is

gcloud compute snapshots delete [SNAPSHOT_NAME]

Creating persistent disk snapshots: https://cloud.google.com/compute/docs/disks/create-snapshots

Restoring and deleting persistent disk snapshots: https://cloud.google.com/compute/docs/disks/restore-and-delete-snapshots

Working with images (e.g. create. an image from a VM or a snapshot, view images, delete an image)

Use images to create managed instance groups and homogeneous boot disks. An image stores all the configuration, permissions, metadata, and data of an instance. The default storage location is the closest multi-region to the source instance. If using an image for replication and cloning, OS and app information unique to the instance needs to be removed before taking the image.

To create an image in the console, go to Compute Engine — Images, then Create Machine Image. Provide a name, select the source instance, and specify where to store the image and if encryption is to be used.

In gcloud the command is

gcloud beta compute machine-images create machine-image-name --source-instance source-instance-name

This also works to create an image from a snapshot or another image

gcloud compute images create image-name --source-disk source-disk -- source-disk-zone zone --family image-family -- storage-location location --force

To create a new instance from an image, go to VM Instance, click New VM Instance from Machine Image, select the machine image and click Continue, then Create.

In gcloud the command is

gcloud beta compute instances create source-instance-name --zone zone --source-machine-image machine-image-name

Images can be deleted by selecting the images in the Images page, then click Delete at the top of the page, or in gcloud using

gcloud compute images delete image-name

Creating machine images: https://cloud.google.com/compute/docs/machine-images/create-machine-images

Creating, deleting, and deprecating custom images: https://cloud.google.com/compute/docs/images/create-delete-deprecate-private-images#gcloud

Working with instance groups (e.g. set autoscaling parameters, assign instance template, create an instance template, remove instance group)

Create an instance template by going to Compute Engine — Instance Templates and clicking on Create Instance Template. Then fill in the details for machine type, GPU, boot disk, service account, API access, firewall, management, security, persistent disk, networking, etc, similar to creating a VM instance.

In gcloud use

gcloud compute instance-templates create example-template-custom --machine-type n1-standard-4 --image-family debian-9 --image-project debian-cloud --boot-disk-size 250GB

To create an Instance Group, go to Compute Engine — Instance Groups, then Create Instance Group. Managed Instance Groups are almost always preferable, as unmanaged instance groups are only used if there are pre-existing, dissimilar VMs that need to be grouped together, e.g. for load balancing. Enter in a location (region/zones), and whether the group is in one zone or multiple zones. Specify the instance template, number of instances, how auto-scaling is done, metrics used to determine if scaling is required, cool down period, min/max number of instances and auto-healing.

In gcloud use

gcloud compute instance-groups managed create instance-group-name --base-instance-name base-name --size size --template instance-template --zone zone

Delete the instance group by selecting the instance group, then clicking Delete, or using

gcloud compute instance-groups managed delete instance-group-name --zone zone

Creating instance templates: https://cloud.google.com/compute/docs/instance-templates/create-instance-templates

Getting, Listing and Deleting Instance Templates: https://cloud.google.com/compute/docs/instance-templates/get-list-delete-instance-templates

Creating managed instance groups: https://cloud.google.com/compute/docs/instance-groups/creating-groups-of-managed-instances

Working with managed instances: https://cloud.google.com/compute/docs/instance-groups/working-with-managed-instances

Working with management interfaces (e.g. Cloud Console, Cloud Shell, GCloud SDK)

First install the GCloud SDK to use gcloud in the command line. Then run gcloud init to initialise some variables, configure gcloud and set up authorisation. This sets up the default configuration, but other configurations can be set up to have different default values for properties such as the project, region and zone.

4.2 Managing Google Kubernetes Engine resources

Viewing current running cluster inventory (nodes, pods, services)

To view all clusters in gcloud

gcloud container clusters list

To view details for a single cluster

gcloud container clusters describe cluster-name

View all node pools in a cluster

gcloud container node-pools list --cluster cluster-name

View details for a single node pool

gcloud container node-pools describe pool-name \ --cluster cluster-name

To view all nodes

kubectl get nodes

To view all pods

kubectl get pods

To view all services

kubectl get services

The Kubernetes Engine Monitoring dashboard can also be used to see clusters, nodes, pods and services, and what the current status is for each of them, within Cloud Monitoring.

Managing cluster: https://cloud.google.com/kubernetes-engine/docs/how-to/managing-clusters

Adding and managing node pools: https://cloud.google.com/kubernetes-engine/docs/how-to/node-pools

Overview of Google Cloud’s operations suite for GKE: https://cloud.google.com/stackdriver/docs/solutions/gke

Browsing the container image repository and viewing container image details

Go to Container Registry in the console to view images that are being stored.

Using gcloud enter

gcloud container images list --repository=[HOSTNAME]/[PROJECT-ID]

In the console, just click on an image to see more details about it. In gcloud, use

gcloud container images list-tags [HOSTNAME]/[PROJECT-ID]/[IMAGE]

Managing images: https://cloud.google.com/container-registry/docs/managing

Working with node pools (e.g. add, edit, or remove a node pool)

To create a node pool in gcloud enter

gcloud container node-pools create pool-name --cluster cluster-name

Resize a node pool using

gcloud container clusters resize cluster-name --node-pool pool-name --num-nodes num-nodes

Upgrade the version of a node pool using

gcloud container clusters upgrade cluster-name

Delete a node pool using

gcloud container node-pools delete pool-name --cluster cluster-name

In the console this is done by going to the Google Kubernetes Engine page, then selecting the cluster, then clicking on Add Node Pool, or the cluster’s edit button.

Adding and managing node pools: https://cloud.google.com/kubernetes-engine/docs/how-to/node-pool

Working with pods (e.g. add, edit, or remove pods)

Pods should be automatically created by the horizontal pod autoscaler based on the resource limit and autoscaling parameters specified for the deployment, and changing the deployment YAML file allows the pods to be edited, or the number to be changed. Use

kubectl get pods

to see the pods that have been created.

Working with services (e.g. add, edit, or remove a service)

All services can be created through YAML files, that specify which service to create, and what network or tags it applies to, and the ports it uses. In the command line you then just have to run

kubectl apply -f my-cip-service.yaml

To edit the services redeploy the updated YAML file.

In the console, go to the deployment details page, then go to Expose. Select the relevant ports, then the type of service to create. Click Expose to create the service for the cluster.

Services can be deleted using

kubectl delete services my-cip-service my-np-service my-lb-service

In the console services can be deleted by just going to Kubernetes Engine, then Services, then select the service and click Delete.

Exposing applications using services: https://cloud.google.com/kubernetes-engine/docs/how-to/exposing-apps

Working with stateful applications (e.g. persistent volumes, stateful sets)

Stateful sets and persistent volumes can be created using kubectl apply with a YAML file that provides the specifications for the objects to be created. They can then be updated using apply again with the updated YAML file. To show information about a particular stateful set use

kubectl get statefulset

and

kubectl describe statefulset

This information can also be seen by going to Kubernetes Engine, and then the Workloads menu.

To see how an update is going use

kubectl rollout status statefulset

To delete a stateful set use

kubectl delete statefulset

A Container Storage Interface (CSI) needs to be installed before persistent volumes can be used in Kubernetes, but Google provides a Compute Engine Persistent Disk driver that’s already available to work with Google Cloud resources. This is specified as an add-on when creating the cluster

gcloud beta container clusters create cluster-name --addons=GcePersistentDiskCsiDriver --cluster-version=version

A storage class is automatically installed with the CSI driver, called standard-rwo, but to check you can run the following command

kubectl get sc

New storage classes can also be defined in YAML files and then created using kubectl create.

After doing this a persistent volume claim can be created in the YAML for a resource, which is then finalised using kubectl create .

Deploying a stateful application: https://cloud.google.com/kubernetes-engine/docs/how-to/stateful-apps

Using the Compute Engine persistent disk CSI driver: https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver

Working with management interfaces (e.g. Cloud Console, Cloud Shell, Cloud SDK)

Covered by the above commands.

4.3 Managing App Engine and Cloud Run resources

Adjusting application traffic splitting parameters

Traffic can be split using IP addresses or HTTP cookies. The IP address method is easier, but the HTTP cookie method is more precise. In the console, go to App Engine then Versions, then select the versions to split traffic between, click Split Traffic, then the method to use for splitting traffic and how much traffic each version should receive.

In gcloud the command is

gcloud app services set-traffic [MY_SERVICE] --splits [MY_VERSION1]=[VERSION1_WEIGHT],[MY_VERSION2]=[VERSION2_WEIGHT] --split-by [IP_OR_COOKIE]

Splitting traffic: https://cloud.google.com/appengine/docs/flexible/python/splitting-traffic

Setting scaling parameters for autoscaling instances

Scaling settings are set in the app.yaml file that is deployed to App Engine. The minimum and maximum number of instances can be set, as well as the metric used to determine whether the system needs to autoscale, which could be request rate, response latency, or other things. The type of scaling, automatic, basic, or manual can also be specified, though basic is only for the standard environment. If an API is used, then the scaling parameters need to be set in the API explorer interface.

Configuring your App with app.yaml: https://cloud.google.com/appengine/docs/flexible/python/configuring-your-app-with-app-yaml

How instances are managed: https://cloud.google.com/appengine/docs/flexible/python/how-instances-are-managed

Working with management interfaces (e.g. Cloud Console, Cloud Shell, Cloud SDK)

See above.

4.4 Managing storage and database solutions

Moving objects between Cloud Storage buckets

In the console, go to the bucket with the object, then click on the three vertical dots at the end of the row for that object, and select Move. Select Browse to find the bucket to move the object to, and then select Move to complete the move.

In gcloud, use

gsutil mv gs://[SOURCE_BUCKET_NAME]/[SOURCE_OBJECT_NAME] gs://[DESTINATION_BUCKET_NAME]/[DESTINATION_OBJECT_NAME]

Renaming, copying, and moving objects: https://cloud.google.com/storage/docs/renaming-copying-moving-objects

Converting Cloud Storage buckets between storage classes

In the console, click on the three vertical dots at the end of the row for the bucket, and select Change Default Storage Class. Then select what the default storage class should be.

In gcloud use

gsutil defstorageclass set [STORAGE_CLASS] gs://[BUCKET_NAME]

The storage class is set for each object in a bucket. To change the storage class of an object gcloud has to be used (can’t use the console)

gsutil rewrite -s [STORAGE_CLASS] gs://[PATH_TO_OBJECT]

Changing the default storage class of a bucket: https://cloud.google.com/storage/docs/changing-default-storage-class

Setting object life cycle management policies for Cloud Storage buckets

Object life cycle management is used to set how long an object stays in one class, and can be used to archive old objects. We need to write a configuration file that specifies the rules that will apply to the bucket, and when the rules are satisfied the specified action is performed.

This can be enabled in the console, command line, or by API. In the console go to GCS, click on the bucket, then click on the Lifecycle tab. Add a rule, and then specify the configuration.

To use the command line, first create a JSON file that has the configuration you want to set, and then use

gsutil lifecycle set [LIFECYCLE_CONFIG_FILE] gs://[BUCKET_NAME]

Managing object lifecycles: https://cloud.google.com/storage/docs/managing-lifecycles

Executing queries to retrieve data from data instances (e.g. Cloud SQL, BigQuery, Cloud Spanner, Cloud Datastore, Cloud Bigtable)

Cloud SQL — First connect to the instance using

gcloud sql connect myinstance --user=root

After this SQL queries can be run normally. This connection can be in Cloud Functions, Cloud Run, App Engine, Compute Engine, Kubernetes Engine, or command line to access the SQL instance from other instances or applications.

Quickstart for Cloud SQL for MySQL: https://cloud.google.com/sql/docs/mysql/quickstart

BigQuery — Queries can be interactive (run as soon as possible), or batch (queued queries that are run as soon as idle resources are available). Queries can be run in the console by selecting the dataset, and then writing and running the query in the code editor window. In the command line, type bq shell to connect to BigQuery, and then write the query, or use bq query ‘<query>’ to run the query. It’s also possible to run queries through the REST API or client libraries.

Using the bq command-line tool: https://cloud.google.com/bigquery/docs/bq-command-line-tool

Running interactive and batch query jobs: https://cloud.google.com/bigquery/docs/running-queries#bigquery-query-cli

Cloud Spanner — It’s possible to run queries in the console similar to BigQuery, or in the command line using gcloud

gcloud spanner databases execute-sql example-db --sql='SELECT SingerId, AlbumId, AlbumTitle FROM Albums'

The REST API can also be used by first using projects.instances.databases.sessions.executeSql, then for the session enter

projects/[PROJECT_ID]/instances/test-instance/databases/example-db/sessions/[SESSION]

In the request body enter the SQL to run, e.g.

{"sql": "SELECT SingerId, AlbumId, AlbumTitle FROM Albums"}

and then execute the SQL.

To use client libraries first get a snapshot object of the database, and then run the execute_sql method of the snapshot:

def query_data(instance_id, database_id):

"""Queries sample data from the database using SQL."""

spanner_client = spanner.Client()

instance = spanner_client.instance(instance_id)

database = instance.database(database_id) with database.snapshot() as snapshot:

results = snapshot.execute_sql(

'SELECT SingerId, AlbumId, AlbumTitle FROM Albums') for row in results:

print(u'SingerId: {}, AlbumId: {}, AlbumTitle: {}'.format(*row))

Quickstart using the console: https://cloud.google.com/spanner/docs/quickstart-console#run_a_query

Getting started with Cloud Spanner using gcloud: https://cloud.google.com/spanner/docs/getting-started/gcloud#query_data_using_sql

Getting started with Cloud Spanner using REST: https://cloud.google.com/spanner/docs/getting-started/rest#query_data_using_sql

Reads: https://cloud.google.com/spanner/docs/reads#python

Cloud Firestore — Queries are run using the client libraries for other languages. First the query is constructed, similar to

query = client.query(kind='Task')

query.add_filter('done', '=', False)

query.add_filter('priority', '>=', 4)

query.order = ['-priority']

and then the query is executed:

query = client.query()

results = list(query.fetch())

Querying and filtering data: https://cloud.google.com/firestore/docs/query-data/queries

Cloud Bigtable — Queries are run using the client libraries for various languages, for example in Python

def read_filter(project_id, instance_id, table_id):

client = bigtable.Client(project=project_id, admin=True)

instance = client.instance(instance_id)

table = instance.table(table_id) rows = table.read_rows(filter_=row_filters.ValueRegexFilter(b"PQ2A.*$"))

for row in rows:

print_row(row)

Reading data: https://cloud.google.com/bigtable/docs/reading-data

Estimating costs of a BigQuery query

The general process to estimate the cost of a query is to determine how much data will be processed, and then enter this amount into the pricing calculator. The volume of data to be processed can be estimated by:

Console — See how much data the query validator says it will process.

bq — In the command line, add the — dry-run flag to the bq query command, and it will show how much data it will process.

API — Submit the API job with dry-run set to true.

Client libraries — In a language such as Python, set the dry_run parameter to True for the QueryJobConfig object, e.g.

job_config = bigquery.QueryJobConfig(dry_run=True, use_query_cache=False)

Pricing calculator — Enter the amount of data that will be processed into the pricing calculator to get the cost (you get 1 TB of on-demand processing for free each month).

Estimating storage and query costs: https://cloud.google.com/bigquery/docs/estimate-costs#console

Backing up and restoring data instances (e.g. Cloud SQL, Cloud Datastore)

Cloud SQL — Backups can be done in the console, gcloud, or REST API, and can be automated. To create a backup in the console, click the Cloud SQL instance, select the Backups tab, then click Create Backup. In gcloud use

gcloud sql backups create --async --instance [INSTANCE_NAME]

To schedule automatic backups in the console, click Manage Automated Backups for the instance, select a time for the backup, and click Save. In gcloud use

gcloud sql instances patch [INSTANCE_NAME] --backup-start-time [HH:MM]

It’s possible to restore from a backup to the same instance, or to a different instance. For the same instance, the data on the instance returns to its state when you took the backup. In the console, select the instance to restore and open Instance details, click Backups, select the backup to use and select Restore. Check the correct instance is chosen, and click Restore. In gcloud use

gcloud sql backups restore [BACKUP_ID] --restore-instance=[INSTANCE_NAME]

When restoring to a different instance, the data on the target is updated to the state of the source from when the backup was taken. In the console select the source instance, open Instance details, and select Backup. Select the backup and click Restore, and then select the target instance, and click Restore. In gcloud use

gcloud sql backups restore [BACKUP_ID] --restore-instance=[TARGET_INSTANCE_NAME] --backup-instance=[SOURCE_INSTANCE_NAME]

Any read replicas need to be deleted before restoring, and then recreated.

Creating and managing on-demand and automatic backups: https://cloud.google.com/sql/docs/mysql/backup-recovery/backing-up

Restoring an instance: https://cloud.google.com/sql/docs/mysql/backup-recovery/restoring

BigQuery — Backup and recovery is handled at the service level. There is also a 7 day history of changes against your tables.

BigQuery for data warehouse practitioners: https://cloud.google.com/solutions/bigquery-data-warehouse#backup_and_recovery

Spanner — Cloud Spanner has options for backup/restore and import/export. In the console select the Instance details, then the backup/restore tab, and in the Backups table click Create. To restore, select the desired backup from the same table, and click Restore.

In gcloud use

gcloud spanner backups create example-db-backup-6 --instance=test-instance --database=example-db --retention-period=1y --async

and

gcloud spanner databases restore --async --destination-instance=test-instance --destination-database=example-db-restored --source-instance=test-instance --source-backup=example-db-backup-6

Backup and Restore with the GCP Console: https://cloud.google.com/spanner/docs/backup/gcp

Backing up and Restoring Databases using gcloud: https://cloud.google.com/spanner/docs/backup/gcloud

Firestore — The managed export and import service is used for backup and restoration, in the console, gcloud, or REST API. A cloud storage bucket needs to be available to store exports in. To export, go to the Firestore/Datastore Entities Export page, set Namespace to All Namespaces, and Kind to All Kinds, then set the destination to the cloud storage bucket, and click Export. In gcloud use

gcloud firestore export gs://bucket-name --async

To import all entities, go to Firestore/Datastore Import, then in the File field select an overall_export_metadata file, set Namespace to All Namespaces, and Kind to All Kinds, and click Import. In gcloud use

gcloud firestore import gs://bucket-name/file-path/file-name.overall_export_metadata --async

Exporting and importing data: https://cloud.google.com/firestore/docs/manage-data/export-import

Bigtable — Backup and restoration can be performed using the console, gcloud, or client libraries. In the console select the instance, then the table to backup. Click Create backup, then enter a unique ID and expiration date for the backup.

In gcloud use

gcloud bigtable backups create BACKUP_ID --instance=INSTANCE_ID --cluster=CLUSTER_ID --table=TABLE_ID --async --expiration-date=EXPIRATION_DATE | --retention-period=RETENTION_PERIOD

To restore a backup in the console, click the instance that has the table to backup, click Backup, then click Restore for the table. Enter a unique ID for the table to be created, then click Restore.

In gcloud use

gcloud bigtable instances tables restore --source-instance=INSTANCE_ID --source-cluster=CLUSTER_ID --source=BACKUP_ID --destination-instance=INSTANCE_ID --destination=NEW_TABLE_ID --async

Managing backups: https://cloud.google.com/bigtable/docs/managing-backups

Reviewing job status in Cloud Dataproc, Cloud Dataflow, or BigQuery

Dataproc — In Cloud Logging select the Dataproc Cluster, then cluster name, then it’s unique ID, and use the logs selector to select google.dataproc.agent. In gcloud you can see a running job’s status using

gcloud dataproc jobs describe job-id --region=region

and

gcloud dataproc jobs list

shows all jobs in the project.

Cloud Logging: https://cloud.google.com/dataproc/docs/guides/logging

Life of a Dataproc job: https://cloud.google.com/dataproc/docs/concepts/jobs/life-of-a-job

Dataflow — Dataflow has a web-based monitoring interface to view and interact with Dataflow jobs, as well as a command line interface. The web interface shows currently running jobs, jobs run in the last 30 days, a graphical representation of each pipeline, details about the job status, links to GCP resources running the pipeline, and any errors or warnings. It can also be used for monitoring individual steps, getting statistical information, and viewing I/O metrics. To access this navigate to the Dataflow section, and then select individual jobs to access monitoring information.

In gcloud you can use the following commands to view jobs, and individual job information

gcloud dataflow jobs list gcloud dataflow jobs describe $JOBID gcloud dataflow logs list $JOBID gcloud dataflow metrics list $JOBID

Using the Dataflow monitoring interface: https://cloud.google.com/dataflow/docs/guides/using-monitoring-intf

Using the Cloud Dataflow command-line interface: https://cloud.google.com/dataflow/docs/guides/using-command-line-intf

BigQuery — Job data can be viewed in the console, command line, API, and client libraries. In the console, click Job history, or query history to view jobs. You can also click Personal history to just view your jobs. On the command line the information about a job is retrieved using

bq --location=location show -j job_id

In a client library use the get_job method of a BigQuery client:

# from google.cloud import bigquery

# client = bigquery.Client()

# job_id = 'bq-job-123x456-123y123z123c' # replace with your job ID

# location = 'us' # replace with your location job = client.get_job(job_id, location=location) # API request # Print selected job properties

print("Details for job {} running in {}:".format(job_id, location)) print(

"\tType: {}

\tState: {}

\tCreated: {}".format(

job.job_type, job.state, job.created

)

)

Job history is kept for the past 6 months.

Managing jobs: https://cloud.google.com/bigquery/docs/managing-jobs

Working with management interfaces (e.g. Cloud Console, Cloud Shell, Cloud SDK)

Covered by the above where applicable. The Cloud Console can be used for most common tasks, otherwise cloud shell/gcloud can be used to execute on the command line.

4.5 Managing networking resources

Adding a subnet to an existing VPC

You first need to create your own VPC (other than the default, which already has a subnet in each region). In the console, select the VPC network to see its network details, then click Add subnet, and provide a name, region, IP address range, and any extra info, then click Add.

In gcloud, use

gcloud compute networks subnets create SUBNET_NAME --network=NETWORK --range=PRIMARY_RANGE --region=REGION

For the REST API, use

POST https://www.googleapis.com/compute/v1/projects/PROJECT_ID/regions/\ REGION/subnetworks

{

"ipCidrRange": "IP_RANGE",

"network": "NETWORK_URL",

"name": "SUBNET_NAME"

}

Using VPC networks: https://cloud.google.com/vpc/docs/using-vpc#add-subnets

Expanding a subnet to have more IP addresses

A subnet can be expanded by modifying the subnet mask of an existing subnet, setting the prefix length to a smaller number in CIDR notation. For an automatically created subnet in an auto-mode network, the broadest prefix is /16. Expanding the primary IP address range can’t be undone.

In the console, click the name of a VPC network, go to its network details page, click the name of the subnet in the Subnets tab to see the Subnet details. Click Edit, then enter a broader CIDR block for the IP address range, and click Save.

In gcloud, use

gcloud compute networks subnets expand-ip-range SUBNET_NAME --region=REGION --prefix-length=PREFIX_LENGTH

In the REST API, use

POST https://www.googleapis.com/compute/v1/projects/PROJECT_ID/regions/\ REGION/subnetworks/SUBNET_NAME/expandIpCidrRange

{

"ipCidrRange": "IP_RANGE"

}

Using VPC networks: https://cloud.google.com/vpc/docs/using-vpc#expand-subnet

Reserving static external or internal IP addresses

A static IP address should be reserved when a virtual machine instance requires a fixed internal or external IP address that won’t change. This can be done by reserving a new internal/external IP address and assigning it to a new VM, or promoting an existing ephemeral IP address to become static.

Static internal IPs reserve the internal IP from the IP range of the subnet, preventing it from being used for automatic allocations, so that the same IP address can be used for the same resource, even if it has to be recreated. The internal IP address of an existing resource can’t be changed, there is a quota on how many internal addresses can be reserved, and it is only supported for VPC networks.

Once a subnet has been created, go to VM instances and create an instance, filling in the required details, then go to Networking, click on Networking interfaces, select the VPC network, choose the subnet, and under Primary Internal IP select Reserve static IP address. Then fill in the remaining details and click Create.

In gcloud use

gcloud compute addresses create ADDRESS_NAME [ADDRESS_NAME..] --region REGION --subnet SUBNETWORK --addresses IP_ADDRESS

To reserve an existing internal IP address, select the instance to change, click Edit, click to edit Network Interfaces, under Internal IP type, select Static, click Done, then Save. In gcloud use

cloud compute addresses create ADDRESS_NAME_1 [ADDRESS_NAME_2..] --addresses IP_ADDRESS_1,[IP_ADDRESS_2,..] --region REGION --subnet SUBNETWORK

A static external IP address is reserved for your project until it’s released. This can be a regional IP address used by VMs or by a network load balancer, or it can be a global IP address which can be used for global load balancers. After the IP address is reserved, it can be assigned during instance creation or to an existing instance. Only one resource at a time can use a static external IP address.

In the console, go to the Reserve a static address page, choose a name, whether it is IPv4 or IPv6 (only global, with global load balancers), choose if it’s regional or global, select a region if required, select a resource to attach it to if available, and click Reserve to reserve the IP.

In gcloud, use

gcloud compute addresses create ADDRESS_NAME [--region REGION | --global ] [--ip-version [IPV4 | IPV6]]

This IP address can be assigned to an instance by providing it as an External IP under the Networking section during creation, or in gcloud using

gcloud compute instances create INSTANCE_NAME --address IP_ADDRESS

To promote an existing external IP address to static, go to the External IP address page, in the Type column change the address type to static for the relevant IP address, provide a name, and click Reserve.

Reserving a static internal IP address: https://cloud.google.com/compute/docs/ip-addresses/reserve-static-internal-ip-address

Reserving a static external IP address: https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address

Working with management interfaces (e.g. Cloud Console, Cloud Shell, Cloud SDK)

See above, these can actions can be performed in all three.

4.6 Monitoring and logging

Creating Stackdriver alerts based on resource metrics

Each alerting policy has conditions that identify when resources are in a state that requires action, and these conditions are continuously monitored. They also have notifications that can be sent through email/SMS/others to let people know the conditions have been met, and documentation that can be included in notifications to help resolve the issue.

When conditions are met, Cloud Monitoring creates and displays an incident in the console, and sends notifications.

In the console, go to Monitoring, then select Alerting, then Create Policy, add conditions for what to track, add a notification channel, and provide any documentation.

In gcloud, alert policies can be expressed in JSON or YAML, which can help to backup and restore policies.

gcloud alpha monitoring policies create --policy-from-file="rising-cpu-usage.json"

Alerts can also be set up from logs, rather than real-time monitoring. Logs-based metrics can be seen in Metrics Explorer, in the Logs-based metrics page in the console. Alerts can be created on logs-based counter metrics by finding the metric to explore, then click on the More menu (three vertical dots), then Create alert from metric. This opens the conditions pane, fill in this information then click Save, which will take you to the Create new alerting policy panel to fill in the notifications and documentation. Give a name for the policy, then click Save.

Managing alerting policies: https://cloud.google.com/monitoring/alerts/using-alerting-ui

Creating charts and alerts: https://cloud.google.com/logging/docs/logs-based-metrics/charts-and-alerts

Creating Stackdriver custom metrics

Time-series data can be written to custom metrics. If we want to keep metrics beyond the retention period then we need to copy the data to Cloud Storage or BigQuery.

The first step is to create a metric descriptor. Cloud Monitoring can do this automatically, or you can use the metricDescriptors.create API. For the Cloud Monitoring method time-series data for the metric is written, and Cloud Monitoring will create a descriptor based on the data. To manually create the descriptor, provide the custom metric type, project, display name, description, kind, value type, and units (optional).

Data can be written to the custom metric type using the timeSeries.create method.

Creating custom metrics: https://cloud.google.com/monitoring/custom-metrics/creating-metrics

Configuring log sinks to export logs to external systems (e.g. on-premises or BigQuery)

Log sinks are used to retain logs for longer periods of time, to analyse logs, or to stream logs to other applications. All logs are sent to the Cloud Logging API, and to export specific logs you need to write a filter that selects them, and choose a destination in Cloud Storage, BigQuery, Pub/Sub, or Cloud Logging. The filter and destination are held in an object called a sink, which can be created in organisations, folders, projects, and billing accounts.

A sink can only export logs that belong to its parent resource, e.g. project/folder. When a log comes in that matches a filter, a copy of the log is written to the export destination.

In the console, go to Logging, then Logs router, and select a project. This should show a table summary of sinks. Click Create Sink, then add a destination service, name, and click Next. Select the destination for the logs and click Next, then build a filter for the logs you want to export, click Next and then Create sink.

In gcloud, use

gcloud alpha logging sinks create sink-name sink-location optional-flags

The console can only create sinks for projects, to create sinks for organisations, folders, or billing accounts, an aggregated sink is required.

Exporting with the Logs Viewer: https://cloud.google.com/logging/docs/export/configure_export_v2

Viewing and filtering logs in Stackdriver

There are two options, Logs Viewer (Classic) and Logs Viewer (Preview). Go to Logging then Logs (Logs Viewer) in the console to see the logs for a project. The drop down menus can be used to select what resource you want to view logs for.

The Preview version can be viewed in the same window by changing the version-picker from Classic to Preview. This window lets you build and save queries to retrieve logs, and view traces.

Logs Viewer (Preview) interface: https://cloud.google.com/logging/docs/view/logs-viewer-interface

Viewing logs (Classic): https://cloud.google.com/logging/docs/view/overview

Viewing specific log message details in Stackdriver

Details for each log can be seen by clicking on the expander arrow for that log in the Logs Viewer, and logs can be downloaded here as well in JSON or CSV format.

Using cloud diagnostics to research an application issue (e.g. viewing Cloud Trace data, using Cloud Debug to view an application point-in-time)

Cloud Trace is a distributed tracing system that collects latency data from your applications. It can be used for finding performance bottlenecks and detecting issues in near-real time. App Engine applications automatically send data to Cloud Trace, but other resources need to be set up. Applications should use OpenTelemetry with the Cloud Trace client library to send data to Cloud Trace.

In the console, trace information can be viewed in Cloud Trace. Overview shows daily analysis, Insights shows performance insights and common problems, and Recent traces shows the most recent traces. The trace list page can also be used to see recent traces, and can be filtered to find a particular trace.

Cloud Debugger can be used to inspect the state of an application, without affecting the app. This is set up differently for each language, and availability on compute environments may depend on language. In general, the broadest possible access scope should be defined to let Cloud Debugger send data, and then IAM should be used to restrict access. The Debugger needs to be installed and configured on the compute environment that you use.

Debugger can take snapshots of variables and the call stack, and inject debug logpoints into a running application. To take a snapshot, navigate to Debugger and then Snapshot, select the source code, and then select the line numbers to take snapshots at. This can be done in gcloud using

gcloud debug snapshots create LOCATION

Once a snapshot has been taken, the variable and call stack details appear in the respective panels in the Debugger window.

Logpoints cause Debugger to log a message every time the code executes the designated line. This can be sent to Cloud Logging for App Engine, though on compute engine the logging agent needs to be used to do this, otherwise logs go to stdout. To add a logpoint navigate to Debugger and then Logpoint, select the source code file, and then the line numbers to add logpoints to. For each one you can add a message, and can add expressions to include values stored in variables. In gcloud use

gcloud debug logpoints create LOCATION MESSAGE

About Cloud Trace: https://cloud.google.com/trace/docs/overview

Trace: Overview: https://cloud.google.com/trace/docs/setup

Debugger Overview: https://cloud.google.com/debugger/docs/setup

Debug Snapshots: https://cloud.google.com/debugger/docs/using/snapshots

Debug Logpoints: https://cloud.google.com/debugger/docs/using/logpoints

Viewing Google Cloud Platform status

This can be viewed from the main homepage/dashboard, to see if there are any service disruptions. The dashboard can also be customised to provide relevant cards and reorganise data. Status can also be viewed at https://status.cloud.google.com/

Working with management interfaces (e.g. Cloud. Console, Cloud Shell, Cloud SDK)

Covered by the above where applicable.

5. Configuring access and security

5.1 Managing identity and access management (IAM)

Viewing IAM role assignments

An IAM policy defines which roles have been assigned to which people. The roles that are available to be granted can be viewed in the console by going to the IAM page, selecting the project, and then choosing a member (user/service account/group). They can also be seen in gcloud using

gcloud iam list-grantable-roles full-resource-name

The role assignments can be viewed by going to the IAM page, which will show the roles assigned to each member. In gcloud use

gcloud iam roles list

The policy bindings can be seen using

gcloud projects get-iam-policy project-id --format=format > filepath

(for projects as an example).

Viewing the grantable roles on resources: https://cloud.google.com/iam/docs/viewing-grantable-roles#iam-grantable-roles-console

Granting, changing, and revoking access to resources: https://cloud.google.com/iam/docs/granting-changing-revoking-access

Assigning IAM roles to accounts or Google Groups

Go to the IAM page in IAM & Admin, then click Add. Specify the email address for the user/service account/Google group, and then select the roles that should be assigned based on least privilege, and click Save. Revoking access can be done on the same page, select the members whose permissions need to be changed, and click Edit. Click Delete for each role to remove, and then Save.

In gcloud a role can be assigned using

gcloud group add-iam-policy-binding resource --member=member --role=role-id

And a role can be revoked using

gcloud group remove-iam-policy-binding resource --member=member --role=role-id

Granting, changing, and revoking access to resources: https://cloud.google.com/iam/docs/granting-changing-revoking-access

Defining custom IAM roles

The first step is to understand what permissions need to be assigned to the custom role, which can be done by looking at what permissions are assigned to predefined roles. This can be done by going to IAM & Admin, and then Roles, then select the role to see what permissions it has. In gcloud this can be done using

gcloud iam list-testable-permissions project-id

Custom roles can be created at the project or organisation level. To create a role, go to Roles, and select the organisation or project. To create a role from scratch, click Create, and then enter a name, title, and description. Click Add Permissions, choose the required permissions, and then Create. To create a custom role from existing roles, select the roles in the Roles window, then click Create Role from Selection.

Add a name, title, and description, and then add or remove any other permissions before clicking Create.

In gcloud this can be done using

gcloud iam roles create

and either providing a YAML file that specifies the permissions and details for the role, or providing this information in flags.

Creating and managing custom roles: https://cloud.google.com/iam/docs/creating-custom-roles

5.2 Managing service accounts

Managing service accounts with limited privileges

To create a service account in the console go to IAM & Admin, and then Service Accounts. Select a project and click Create Service Account, then provide a name, ID, description, specify what role(s) the service account should have, and click Save.

In gcloud use

gcloud iam service-accounts create sa-name --description="sa-description" --display-name="sa-display-name"

All created service accounts will be visible on the Service Accounts page, or in gcloud use

gcloud iam service-accounts list

To update a service account, select the More (three vertical dots) button for that account, then Edit, to update the name and description.

gcloud iam service-accounts update sa-name@project-id.iam.gserviceaccount.com --description="updated-sa-description" --display-name="updated-display-name"

Service accounts can be disabled by selecting the service account, and then selecting Disable service account (under Service account status). In gcloud use

gcloud iam service-accounts disable sa-name@project-id.iam.gserviceaccount.com

The opposite of this can be done to enable service accounts.

Service accounts should be disabled before deleting them to ensure that removing them will not negatively impact other resources. They can be deleted in the console by selecting the service account and clicking Delete, or in gcloud using

gcloud iam service-accounts delete sa-name@project-id.iam.gserviceaccount.com

Creating and managing service accounts: https://cloud.google.com/iam/docs/creating-managing-service-accounts

Assigning a service account to VM instances

First create a service account, and then the service account can be assigned to new or existing VM instances. Each VM can only be assigned to one service account, but a service account can have many VM instances.

When creating an instance in the console go to Compute Engine, then Create Instance, and in the Identity and API Access section choose the relevant service account, and then create the instance. This can also be applied to instance templates.

To do the same thing in gcloud use the service account email address and provide the access scopes

gcloud compute instances create [INSTANCE_NAME] --service-account [SERVICE_ACCOUNT_EMAIL] --scopes [SCOPES,...]

To update the service account of an existing instance you first need to stop the instance in the VM instances page. Then click Edit, scroll down to the service account section, and choose the desired service account. Access scopes can also be changed if required, then click Save, and restart the instance.

Creating and enabling service accounts for instances: https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances

Granting access to a service account in another project

To do this the service account needs to have been created in the other project (Project B), and you need to know its ID or email address. In this project (Project A), go to IAM & Admin and then IAM, and click Add. Add the email address for the service account in Project B, and assign it the required roles in Project A. This ensures that the service account can access resources in Project A, without affecting its access in the other project.

5.3 Viewing audit logs for project and managed services

Cloud Audit Logs provides logs for Admin Activity, Data Access, and System Events for projects, folders, and the organisation. Admin Activity records actions that modify the configuration or metadata of resources, such as creating instances, and these are always enabled. Data Access logs record calls to read the configuration or metadata of resources, or that create, modify, or read user-provided resource data. They are disabled by default. System Event logs record Google Cloud administrative actions that modify the configuration of resources, which are not the result of user behaviour. These are always enabled.

Abbreviated audit log entries can be seen in the Activity page for the project or organisation. Select the project or organisation to view logs for, and then filter as required.

The full logs can be seen by going to Cloud Logging, then access the new Logs Viewer, then choose the project. In the Query Builder select the resource you want to see logs for, and then choose one of the three audit log types. This can be done in gcloud using

gcloud logging read "logName : projects/project-id/logs/cloudaudit.googleapis.com" --project=project-id

Cloud Audit Logs: https://cloud.google.com/logging/docs/audit