GCP Certification Series: 3.4 Deploying and implementing data solutions
By: Prashanta Paudel, Prashantapaudel.Com.Np
Tags: Deployment, Data Solutions, Google Cloud Platform, Gcp Certification

Objectives

Clone or download the sample app.

Build the app and run it on your local machine.

Deploy the app to App Engine.

Walk through the sample code.

Learn how the app stores structured data.

Learn how the app stores binary data in Google Cloud Storage.

Learn how the app authenticates users.

Learn how the app creates event logs that are visible in the Google Cloud Platform Console.

This tutorial explores the Bookshelf app in detail and discusses how each feature of the app is implemented using familiar technologies and services provided by Cloud Platform.

Build a bookshelf application that runs in app engine but saves the database in Cloud Datastore.

Bookshelf app

A kind in Cloud Datastore is roughly equivalent to a table in a relational database. Kinds are used to categorize entities for queries.

Data objects in Google Cloud Datastore are known as entities. An entity has one or more named properties (which are like fields in a relational database). Each property can have one or more values. Entities are identified by a unique key and are akin to rows in a relational database. Entities of the same kind need not have the same properties. These unique characteristics imply a different way of designing and managing data to take advantage of the ability to scale automatically.

Cloud Datastore also organizes data into entity groups. An entity group consists of a root entity and all of its descendants. Applications typically use entity groups to organize highly related data to achieve strong consistency and to enforce transactionality. For example, an application could use an entity group to store data about one product, or one user profile.

The Bookshelf data model for Cloud Datastore is illustrated by the following diagram. Note that the simple data model used in Bookshelf does not make use of entity groups.

Before you begin

Create a new GCP project, and then create an App Engine application and enable billing in that project.

Enable the Cloud Datastore, Cloud Pub/Sub, Cloud Storage JSON, Stackdriver Logging, Google+, and Google Cloud SQL APIs.

Verify that your default project is correct:

gcloud config list

If the project ID listed in the output is not the project that you intended to use for this tutorial, set the project by entering this command:

gcloud config set project [YOUR_PROJECT_ID]

where [YOUR_PROJECT_ID] is the ID of the project you created or chose to use for this tutorial.

Clone the sample repository:

Alternatively, you can download the sample as a zip file and extract it.

Now, here comes main steps:

Step 1: Clone repository from github

git clone https://github.com/GoogleCloudPlatform/getting-started-php.git

Cloning into 'getting-started-php'...

remote: Enumerating objects: 7, done.

remote: Counting objects: 100% (7/7), done.

remote: Compressing objects: 100% (7/7), done.

remote: Total 2143 (delta 0), reused 2 (delta 0), pack-reused 2136

Receiving objects: 100% (2143/2143), 1.04 MiB | 0 bytes/s, done.

Resolving deltas: 100% (1419/1419), done. prashantagcppaudel@cloudshell:~ (pfour-221109)$Cloning into 'getting-started-php'...remote: Enumerating objects: 7, done.remote: Counting objects: 100% (7/7), done.remote: Compressing objects: 100% (7/7), done.remote: Total 2143 (delta 0), reused 2 (delta 0), pack-reused 2136Receiving objects: 100% (2143/2143), 1.04 MiB | 0 bytes/s, done.Resolving deltas: 100% (1419/1419), done.

Step 2: Goto getting started folder and 2-structured-data folder

$ cd getting-started-php/

$ cd 2-structured-data/

Step 3: copy settings.yaml.dist into settings.yaml

$ cp config/settings.yml.dist config/settings.yml

Step 4: Again goto config folder

$ cd config/

Step 5: Edit settings .yml as follows

Step 6: Install composer

$ composer install

Loading composer repositories with package information

Installing dependencies (including require-dev) from lock file

Package operations: 52 installs, 0 updates, 0 removals

- Installing firebase/php-jwt (v4.0.0): Downloading (100%)

- Installing rize/uri-template (0.3.2): Downloading (100%)

- Installing psr/http-message (1.0.1): Downloading (100%)

- Installing psr/log (1.0.2): Downloading (100%)

- Installing monolog/monolog (1.23.0): Downloading (100%)

- Installing guzzlehttp/psr7 (1.4.2): Downloading (100%)

- Installing guzzlehttp/promises (v1.3.1): Downloading (100%)

- Installing guzzlehttp/guzzle (6.3.0): Downloading (100%)

- Installing psr/cache (1.0.1): Downloading (100%)

- Installing google/auth (v0.11.1): Downloading (100%)

- Installing google/cloud (v0.21.1): Downloading (100%)

- Installing symfony/routing (v3.0.9): Downloading (100%)

- Installing symfony/polyfill-mbstring (v1.7.0): Downloading (100%)

- Installing symfony/http-foundation (v3.0.9): Downloading (100%)

- Installing symfony/event-dispatcher (v3.0.9): Downloading (100%)

- Installing symfony/debug (v3.4.4): Downloading (100%)

- Installing symfony/http-kernel (v3.0.9): Downloading (100%)

- Installing pimple/pimple (v1.1.1): Downloading (100%)

- Installing silex/silex (v1.3.6): Downloading (100%)monolog/monolog suggests installing php-amqplib/php-amqplib (Allow sending log messages to an AMQP server using php-amqplib)

monolog/monolog suggests installing php-console/php-console (Allow sending log messages to Google Chrome)

monolog/monolog suggests installing rollbar/rollbar (Allow sending log messages to Rollbar)

monolog/monolog suggests installing ruflin/elastica (Allow sending log messages to an Elastic Search server)

monolog/monolog suggests installing sentry/sentry (Allow sending log messages to a Sentry server)

google/cloud suggests installing google/gax (Required to support gRPC)

google/cloud suggests installing google/proto-client-php (Required to support gRPC)

google/cloud suggests installing james-heinrich/getid3 (Allows the Google Cloud Speech client to determine sample rate and encoding of audio inputs)

symfony/routing suggests installing doctrine/annotations (For using the annotation loader)

symfony/routing suggests installing symfony/config (For using the all-in-one router or any loader)

symfony/routing suggests installing symfony/dependency-injection (For loading routes from a service)

symfony/routing suggests installing symfony/expression-language (For using expression matching)

symfony/event-dispatcher suggests installing symfony/dependency-injection

symfony/http-kernel suggests installing symfony/class-loader

symfony/http-kernel suggests installing symfony/config

symfony/http-kernel suggests installing symfony/dependency-injection

symfony/http-kernel suggests installing symfony/finder

symfony/http-kernel suggests installing symfony/var-dumper

behat/mink suggests installing behat/mink-selenium2-driver (slow, but JS-enabled driver for any app (requires Selenium2))

behat/mink suggests installing behat/mink-zombie-driver (fast and JS-enabled headless driver for any app (requires node.js))

symfony/console suggests installing symfony/lock

sebastian/global-state suggests installing ext-uopz (*)

phpunit/phpunit-mock-objects suggests installing ext-soap (*)

phpunit/php-code-coverage suggests installing ext-xdebug (>=2.2.1)

phpunit/phpunit suggests installing phpunit/php-invoker (~1.1)

Generating autoload files

Step 7: Deploy App in App Engine

$ gcloud app deploy

3acd409e5aea: Layer already exists

363b7c2debba: Pushed

ed59ef025e1e: Layer already exists

7420484a2283: Pushed

5bdaf3588472: Layer already exists

03fce3219a27: Layer already exists

56f56dd1c2db: Layer already exists

2f4254df2a74: Layer already exists

3bfe3bd8f9e5: Pushed

89d0200fdea0: Layer already exists

f7fb3b0f4fc6: Layer already exists

d464d4f089a3: Layer already exists

62f57a66e54a: Layer already exists

1c1171b5f438: Layer already exists

b00f214424ac: Layer already exists

96e208368397: Layer already exists

8c468b8e19fa: Layer already exists

b056439591da: Layer already exists

84ff92691f90: Layer already exists

latest: digest: sha256:401c366752609e012efa055233becd40dce65dcb6ad70a23e820a2370fd5f50f size: 4911

DONE

----------------------------------------------------------------------------------------------------------------------------------- Updating service [default] (this may take several minutes)...done.

Setting traffic split for service [default]...done.

Deployed service [default] to [https://pfour-221109.appspot.com] You can stream logs from the command line by running:

$ gcloud app logs tail -s default To view your application in the web browser run:

$ gcloud app browse

Step 8: Now check the website on https://pfour-221109.appspot.com.

FINISHED !

The information is stored in datastore as folows

========================================

Google BigQuery

Big Query

BigQuery is a fast, highly scalable, cost-effective, and fully managed cloud data warehouse for analytics, with built-in machine learning.

BigQuery is Google’s serverless, highly scalable, enterprise data warehouse designed to make all your data analysts productive at an unmatched price-performance. Because there is no infrastructure to manage, you can focus on analyzing data to find meaningful insights using familiar SQL without the need for a database administrator.

Analyze all your data by creating a logical data warehouse over managed, columnar storage, as well as data from object storage and spreadsheets. Build and operationalize machine learning solutions with simple SQL. Easily and securely share insights within your organization and beyond as datasets, queries, spreadsheets, and reports.

BigQuery allows organizations to capture and analyze data in real time using its powerful streaming ingestion capability so that your insights are always current, and it’s free for up to 1 TB of data analyzed each month and 10 GB of data stored.

Reference: https://cloud.google.com/bigquery/

Reference: https://cloud.google.com/bigquery/

Reference: https://cloud.google.com/bigquery/

Reference: https://cloud.google.com/bigquery/

Reference: https://cloud.google.com/bigquery/

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — -

BigQuery Example

Run a Query on public dataset

First of all you must have your GCP account and billing setup ready.

Now goto BigQuery in Big Data.

You will get either a old UI or New UI based on when you open as new UI is on beta.

BigQuery

Basically speaking, this UI and CLI tool is the engine to process all your databases for Analytical purpose.

Most basic task is to load data into the storage or in Bigquery itself and run query on it.

So, First download public data from the link http://www.ssa.gov/OACT/babynames/names.zip

Unzip this file and check the datas. Select some year and copy the txt file to desktop

Again goto BigQuery Web UI and navigation to Resources section, click your project name.

On the right side, in the details panel, click Create dataset. On the Create dataset page:

For Dataset ID , enter babynames .

, enter . For Data location, choose United States (US). Currently, the public datasets are stored in the US multi-region location. For simplicity, you should place your dataset in the same location.

Leave all of the other default settings in place and click Create dataset.

Load the data into a new table

Next, load the data into a new table.

In the navigation panel, in the Resources section, click the babynames dataset that you just created. On the right side, in the details panel, click Create table. Use the default values for all settings unless otherwise indicated. On the Create table page:

For Source Data , click Empty table and choose Upload .

, click and choose . For Select file , click Browse , navigate to the yob2014.txt file and click Open .

, click , navigate to the file and click . For File format , click Avro and choose CSV .

, click and choose . For Destination table , enter names_2014 .

, enter . In the Schema section, click the Edit as text toggle and paste the following schema definition in the box.

section, click the toggle and paste the following schema definition in the box. name:string,gender:string,count:integer

Click Create Table. Wait for BigQuery to create the table and load the data. While BigQuery loads the data, a (1 running) string displays beside the job history in the navigation panel. The string disappears after the data is loaded.

Preview the table

After the (1 running) string disappears, you can access the table. To preview the first few rows of the data:

Select babynames > names_2014 in the navigation panel. In the details panel, click the Preview tab.

Query the table

Now that you’ve loaded data into a table, you can run queries against it. The process is identical to the previous example, except that this time, you’re querying your table instead of a public table.

If necessary, click the Compose new query button. Unless you hid the query window previously, it should still be visible. Copy and paste the following query into the query text area. This query retrieves the top 5 baby names for US males in 2014.

SELECT

name, count

FROM

`babynames.names_2014`

WHERE

gender = ‘M’

ORDER BY count DESC LIMIT 5

Click Run query. The results are displayed below the query window.

Query result

======================================

Cloud Spanner

Cloud Spanner

Cloud Spanner is a fully managed, mission-critical relational database system. It has special properties like

High Availability Transactional consistency global scale

With traditional relational semantics like schemas, ACID transaction, SQL with automatic and synchronous replication cloud spanner is the only product of its kind in the market.

Cloud spanner is first horizontally scalable. strongly consistent, relational database service.

Cloud Spanner is the only enterprise-grade, globally-distributed, and strongly consistent database service built for the cloud specifically to combine the benefits of a relational database structure with non-relational horizontal scale. This combination delivers high-performance transactions and strong consistency across rows, regions, and continents with an industry-leading 99.999% availability SLA, no planned downtime, and enterprise-grade security. Cloud Spanner revolutionizes database administration and management and makes application development more efficient.

In today’s always-on, globally-distributed world, IT and developer efficiency, measured in the app downtime and time to market, is one of an organization’s most precious resources. The challenge of efficiently managing app database backends while at the same time giving developers the tools they need to build efficiently was previously a challenge.

References: https://cloud.google.com/spanner/

Use Cases

References: https://cloud.google.com/spanner/

References: https://cloud.google.com/spanner/

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Example : Cloud Spanner

Cloud spanner is only used when you need better scalability and higher availaility.

Creating an Instance

In Cloud Spanner compute nodes and storage are separated to enable scaling each independently. In addition, your nodes and data are replicated across multiple zones or even regions for high availability.

Further , I created table and column in database

Scaling an Instance

For each node you request for your instance you get compute resources in all zones and regions depending on your selected configuration. There is no need for difficult planning and forecasting on how many resources to provision upfront. You can just change the number of instance nodes any time you need more or fewer.

It is easy to scale your Cloud Spanner Instance!

Reading and Writing Data

Reading and writting can be done from client machines and through API.

Insert Data

Query Data

Query Data

========================================

CLOUD PUB/SUB

Ingest event streams from anywhere, at any scale, for simple, reliable, real-time stream analytics

Deliver event data wherever you need it

Cloud Pub/Sub is a simple, reliable, scalable foundation for stream analytics and event-driven computing systems. As part of Google Cloud’s stream analytics solution, the service ingests event streams and delivers them to Cloud Dataflow for processing and BigQuery for analysis as a data warehousing solution. Relying on the Cloud Pub/Sub service for delivery of event data frees you to focus on transforming your business and data systems with applications such as:

Real-time personalization in gaming

Fast reporting, targeting and optimization in advertising and media

Processing device data for healthcare, manufacturing, oil and gas, and logistics

Syndicating market-related data streams for financial services

Build multi-cloud and hybrid applications on open architecture

Syndicate data across projects and applications running on other clouds, or between cloud and on-premises apps. Cloud Pub/Sub easily fits in your existing environment via efficient client libraries for multiple languages, open REST/HTTP and gRPC service APIs, and an open source Apache Kafka connector.

Scale responsively and automatically

Scale to hundreds of millions of messages per second and pay only for the resources you use. There are no partitions or local instances to manage, reducing operational overhead. Data is automatically and intelligently distributed across data centers over our unique, high-speed private network.

Bring reliability and security tools to real-time apps

Use Cloud Pub/Sub to simplify scalable, distributed systems. All published data is synchronously replicated across availability zones to ensure that messages are available to consumers for processing as soon as they are ready. Fine-grained access controls allow for sophisticated cross-team and organizational data sharing. And end-to-end encryption adds security to your pipelines.

Ref: https://cloud.google.com/pubsub/

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — -

Example : Cloud Pub/Sub

Starting cloud pub/sub

Cloud Pub/Sub Tutorial

This simple tutorial demonstrates writing, deploying, and triggering a Background Cloud Function with a Cloud Pub/Sub trigger. To learn more about Cloud Pub/Sub, see the Cloud Pub/Sub documentation.

Note: You can also use HTTP-triggered functions to listen to Pub/Sub push subscriptions. This allows a single Cloud Function to subscribe to multiple topics.

Objectives

Write and deploy a Background Cloud Function.

Trigger the function by publishing a message to a Cloud Pub/Sub topic.

Preparing the application

Clone the sample app repository to your local machine:

PYTHON (BETA)

git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git

Alternatively, you can download the sample as a zip file and extract it Change to the directory that contains the Cloud Functions sample code for accessing Cloud Pub/Sub:

PYTHON (BETA)

cd python-docs-samples/functions/helloworld/

Take a look at the sample code:

PYTHON (BETA)

def hello_pubsub(data, context):

“””Background Cloud Function to be triggered by Pub/Sub.

Args:

data (dict): The dictionary with data specific to this type of event.

context (google.cloud.functions.Context): The Cloud Functions event

metadata.

“””

import base64

“””Background Cloud Function to be triggered by Pub/Sub. Args: data (dict): The dictionary with data specific to this type of event. context (google.cloud.functions.Context): The Cloud Functions event metadata. “”” import base64 if ‘data’ in data:

name = base64.b64decode(data[‘data’]).decode(‘utf-8’)

else:

name = ‘World’

print(‘Hello, {}!’.format(name))

Deploying the function

To deploy the function with a Cloud Pub/Sub trigger, run the following command in the directory where the sample code is located:

PYTHON (BETA)

gcloud functions deploy hello_pubsub --runtime python37 --trigger-resource YOUR_TOPIC_NAME --trigger-event google.pubsub.topic.publish

where YOUR_TOPIC_NAME is the name of the Cloud Pub/Sub topic to which the function will be subscribed.

Note: The function deployment creates the specified Cloud Pub/Sub topic for you. You can also specify an existing topic in your project to subscribe to.

Triggering the function

Publish a message to your Cloud Pub/Sub topic. In this example, the message is a name that the function will include in a greeting:

gcloud pubsub topics publish YOUR_TOPIC_NAME --message YOUR_NAME

Replace YOUR_TOPIC_NAME with the name of your Cloud Pub/Sub topic, and YOUR_NAME with an arbitrary string. Check the logs to be sure the executions have completed:

gcloud functions logs read — limit 50