Learnings from Streaming 25 Billion Events to Google BigQuery
By: Aride Chettali
Tags: Data, Bigquery, Big Data, Google Cloud Platform, Pipeline

Learnings

Batching:

BigQuery streaming API gives an option to batch your rows into a single request and call the BigQuery endpoint. I tried with various batch sizes for my use case and observed when batch size was 500 records (1.25 MB) latency was around 450ms — 650ms and when the batch size was 1000 records the latency was around 1100ms — 1500ms. I chose 500 as batch size as it gave a better latency for our use case.

Streaming Insertion Quota Limit:

The default quota for streaming maximum bytes per second is 1GB per GCP project. Any ingestion above this limit would result in BigQueryException with quotaExceeded Error. When I ramped our ingestion traffic near 60% I hit this limit and requests started to fail with the above Exception.

The way to handle this error is to increase the project level insertion quota as per your need by creating a ticket to Google and implement retry logic to retry the same request on every error to avoid losing the failed record in the final table.

This quota usage can be monitored with the below BigQuery SQL and adjust the quota well before you reach the quota limit.

SELECT start_timestamp, SUM(total_requests) AS total_requests, SUM(total_rows) AS total_rows, SUM(total_input_bytes) AS total_input_bytes, SUM(IF(error_code IN (“QUOTA_EXCEEDED”, “RATE_LIMIT_EXCEEDED”), total_requests, 0)) AS quota_error, SUM(IF(error_code IN (“INVALID_VALUE”, “NOT_FOUND”, “SCHEMA_INCOMPATIBLE”, “BILLING_NOT_ENABLED”, “ACCESS_DENIED”, “CUNAUTHENTICATED”), total_requests, 0)) AS user_error, SUM(IF(error_code IN (“CONNECTION_ERROR”,“INTERNAL_ERROR”), total_requests, 0)) AS server_error, SUM(IF(error_code IS NULL, 0, total_requests)) AS total_error, FROM `region-us`.INFORMATION_SCHEMA.STREAMING_TIMELINE_BY_PROJECT GROUP BY start_timestamp ORDER BY 1 DESC

The quota is not calculated based on your actual streaming payload to BigQuery but based on the streaming payload + metadata. Your payload is sent inside a JSON and quota is computed based on the received bytes at the BigQuery end. So the additional tags from JSON are also counted in the current quota calculation.

Enabling Deduplication:

In my PoC, I observed that during streaming ingestion data gets duplicated, and duplicate records are found in the final table.

BigQuery is also providing a way to enable deduplication. There are two ways you can create “RowToInsert” object from the BiqQuery client. One without deduplication enabled and one with deduplication enabled.

// 1. insertion without dedupe

RowToInsert row = InsertAllRequest.RowToInsert.of(content); // 2. insertion with dedupe

RowToInsert row = InsertAllRequest.RowToInsert.of(uniqueInsertId,content);

When I ingested without enabling “dedupe” one record is getting duplicated for every 500,000 records ingested.

When I enabled “dedupe” only one record is getting duplicated for every 5 million records ingested. Enabling deduplication does not guarantee 100% duplication removal rather it is only the best effort to remove duplicates.

BigQueyr uses “uniqueInsertId” present in every record to identify and remove the duplicates. And this deduplication is performed against the last 10 minutes of the received records.

Caching on Streaming Ingestion Table:

BigQuery provides basic caching for the batch ingested tables. When you run a duplicate query, BigQuery attempts to reuse cached results. To retrieve data from the cache.

However, if the data is ingested to the table through streaming then Bigquery does not provide any caching on it.

Higher Latency Querying Streaming Table:

I observed higher latency when querying the streaming table compared to the batch ingested table even when the amount of data scanned from the streaming table is lesser.

The reason for this higher latency in the case of the streaming table is due to the requirements to scan the streaming buffer for the data. Streaming buffer is the place the ingested records are stored initially and then the records are flushed to permanent BigQuery Capacitor columnar storage.

Capacitor Columnar storage is optimized for read-heavy operations and BigQuery Streaming buffer is optimized for write-heavy operations. When you perform read operations from a streaming table you end up reading data from write optimized streaming buffer and that's the exact reason for higher latency. The same is reflected in the above picture, the totalPartitionsProcessed is 2 for the Streaming table but it is 1 for the Batch ingested table. You shouldn’t worry about this additional latency as you are processing the latest data included from the streaming buffer. The additional latency can be 15–30 seconds depending upon the amount of data that you are querying.

The streaming buffer can take 10–90 minutes to flush the data to the permanent Columnar storage. In the latest version of the BigQuery backend (v2) this is reduced to 2–3 minutes. As of writing this blog the default version of the BigQuery backend is v1.