Data storage options on Google Cloud
By: Avanish Chauhan, Senior Java Developer
Tags: Google, Google Cloud Platform, Data Storage, Cloud Storage, Luxoft

This is Avanish Chauhan, having 8+ years of experience in Backend technologies like Java, GoLang and RubyOnRails. For the last one and half years, I have been working with Luxoft as Senior Software Developer/Scrum Master. While working with Luxoft, I got an opportunity to work on different tools and technologies, working on Google Cloud Platform is one of them.

While migrating an application from on-premises to cloud or developing an application on cloud-only, one of the biggest challenges is to choose appropriate data storage and corresponding strategy. Why is it so important and how to deal with it?

In this article, I am going to cover why choosing an appropriate data storage is really important and how to handle it while working with Google Cloud or Google Cloud Platform.

Importance of choosing the right data storage

1. Data storage is costly: While storing data on any cloud you have to pay a good amount of money, Google cloud charges differently for the various data storage mechanism. You have to be very careful while opting for any data storage.

2. Size of data generated by the application: Data generated by application varies from one application to another, an application like Twitter, LinkedIn, Instagram, Facebook and such similar social media application generated TBs of data per week, so if you are developing or migrating such application to Google Cloud, make sure to keep this in mind.

3. Usage of generated data: Few companies use generated data for analysis, data warehousing, trend analysis and some others can use it to just to keep it stored in order to recover that data in case of any disaster. So, it’s an important point to consider while choosing data storage on Google Cloud.

4. Data Availability and retention policy: In Google Cloud, you can not expect a similar type of data available for all the data storages. Few storages keep data only for 30 days, some other for 1 year, so make a note of this as well.

5. Source of data creation: Data generated by IoT devices can not be handled properly by the data storage services which are meant to store data generated from the mobile application or desktop applications.

6. Type of database required: Google cloud comes with different options while choosing data storage, you can choose SQL, NoSQL, Transactional, database with horizontal spanning capacity and many more.

Data storage options provided by Google Cloud

Google cloud offers a range of storage options which can be used to fulfil the needs of any type of user. Depending on the type of your application you can choose any storage option from this list:

1. Cloud Storage

2. Cloud SQL

3. Cloud Spanner

4. Cloud Datastore

5. Cloud Bigtable

6. Big query

Let me put this in a very simple way to understand:

Storage options as per the type of data:

Relational SQL for OLTP data: Cloud SQL, Cloud Spanner, and BigQuery are the best candidate if your applications deal with Relational SQL for OLTP data.

2. As per the transaction support:

Cloud datastore, Cloud SQL, and Cloud Spanner supports transactions, Bigtable supports only single-row transaction, while Cloud storage and BigQuery doesn’t support transactions.

3. Support for Complex queries:

All relational databases like Cloud SQL, Cloud Spanner and BigQuery supports complex queries, while non relational storages like Cloud Datastore, Bigtable and Cloud storage have no option for complex queries.

4. As per the capacity:

Cloud SQL — Up to 10 Terabytes

Cloud Datastore — Terabytes

Cloud Spanner, Cloud Storage, Bigtable, BigQuery- Petabytes of data

5. When to choose which data storage option?

Cloud Datastore is best suited for Semi-structured, application data and durable key-value data.

If your application deals with “Flat” data and heavy read and write operations, and analystical data, go for Cloud Bigtable.

For Object storage, choose Cloud Storage, which is also best fit for Structured and Unstructured binary or object data.

Having web-application which incorporates multiple Web frameworks? choose Cloud SQL.

If you are looking for a SQL database and planning to scale your application globally in near by future as your business is expanding and number of users is increasing, then Cloud Spanner is the best choice.

For intercative querying and offline analysis of the data, select BigQuery storage option.

6. Best Use cases:

Cloud datastore: App engine applications

Cloud Bigtable: Data from IoT devices, AdTech and Financial date.

Cloud Storage: To store large images, media files and backup files.

Cloud SQL: For storing User credentials and for order management applications.

Cloud Spanner: To scale your application globally so that high I/O and global consistency can be achieved.

BigQuery: For data analysis and data warehousing.

Storage classes provided by Cloud storage:

Google Cloud storage provided four types of storage classes:

Multi-regional: For most frequently accessed data globally, SLA of 99.95%, the high price per GB stored. Regional: For data accessed frequently within a region, SLA of 99.90%, Data storage Cheaper than multi-regional. Nearline: For data accessed less than once per month, SLA of 99.00%, high data transfer price as compared to Regional but lesser data storage price. Coldline: where data is accessed less than once in a year, SLA of 99.00, higher data transfer prices as compared to Nearline but lesser data storage price.

I hope this article gives you a fair idea of how to choose a storage option from Google Cloud data storage options.

Important tip: If you are preparing for Google Cloud Certified Associate Cloud Engineer, cloud storage options is one of the areas from which you can expect multiple questions, so be well prepared.

“Happy Learning”