Package: bigtableql
BigtableQLBigtableQL provides a SQL Query Layer forGoogle Cloud Bigtable.Use CasesCloud Bigtable is Google's fully managed NoSQL Big Data database service. Each table contains rows and columns. Each row/column intersection can contain multiple cells. Each cell contains a unique timestamped version of the data for that row and column. Thus Bigtable is often used to store time series data.BigtableQL provides a SQL query layer to run aggregation query on Bigtable.Quick Startpip install bigtableqlUsing theweather balloon example datashown inSingle-timestamp unserializedschema designRow key                         pressure    temperature humidity    altitude
us-west2#3698#2021-03-05-1200   94558       9.6         61          612
us-west2#3698#2021-03-05-1201   94122       9.7         62          611
us-west2#3698#2021-03-05-1202   95992       9.5         58          602
us-west2#3698#2021-03-05-1203   96025       9.5         66          598
us-west2#3698#2021-03-05-1204   96021       9.6         63          624After initialize the clientfrom bigtableql.client import Client
# config follows offical python bigtable client
client = Client(config)

client.register_table(
    "weather_balloons",
    instance_id="INSTANCE_ID",
    column_families={
        "measurements": {
            "only_read_latest": True,
            "columns": {
                "pressure": int,
                "temperature": str,
                "humidity": int,
                "altitude": int
            }
        }
    }
)we are able to calculate average pressure of the period byclient.query("measurements", """
SELECT avg(pressure) FROM weather_balloons
WHERE
  "_row_key" BETWEEN 'us-west2#3698#2021-03-05-1200' AND 'us-west2#3698#2021-03-05-1204'
""")Or with row key decompositionclient.register_table(
    xxx,
    row_key_identifiers=["location", "balloon_id", "event_minute"],
    row_key_separator="#"
)

client.query("measurements", """
SELECT balloon_id, avg(pressure) FROM weather_balloons
WHERE
  location = 'us-west2'
  AND balloon_id IN ('3698', '3700')
  AND event_minute BETWEEN '2021-03-05-1200' AND '2021-03-05-1204'
GROUP BY 1
""")The output ofqueryis list ofpyarrow.RecordBatch. It can be easily convert to python dictionary (to_pydict) and pandas dataframe (to_pandas).Group by TimeEach cell in Bigtable have a timestamp.SELECT "_timestamp"will return a float number, represent seconds since Unix epoch. Following is an example to select 5 minutes intervalSELECT to_timestamp_seconds(cast(floor("_timestamp" / 600) * 600 as bigint)) as intervalAlternativeGoogle BigQuery external data sourceHowever, as of 2022-01, itonly supports "us-central1" and "europe-west1" regiononly supports query with "rowkey"by default can run up to4 concurrent queriesagainst Bigtable external data sourceRoadmapSQL✅ Insert Into✅ Select *✅ Select column(s)✅ Filter (WHERE): "=", "IN", "BETWEEN", ">", ">=", "<", "<="✅ GROUP BY✅ ORDER BY✅ HAVING✅ LIMIT✅ Aggregate (e.g. avg, sum, count)✅ AND✅ Alias✅ Cast✅ Common Math FunctionsCommon Date/Time FunctionsOR ???Join ???General✅ Partition Pruning✅ Projection pushdown✅ Predicate push down (Value range)Limit Pushdown ???Limitationfor row key encoding, only string is supportedfor single/composite row key, identifiers supports "=" and "IN". Additionally, last identifier also supports "BETWEEN".for qualifiers, only string and integer (64bit BigEndian encoding) value are supportedsubqueries and common table expressions are not supportedTechnical DetailsBigtableQL depends onsqloxideandsqlparser-rs: SQL parserpython-bigtable: offical python bigtable clientdatafusion-python: in memory query enginepyarrow: in memory columnar store / dataframe