Repository: opendatacube/datacube-core

Issue:
Datacube Database Federation and ETL using timestamps
The append only nature of Datacube should have made ETL of changes easy with just primary keys. However updates are more difficult to track and is a relatively new change in Datacube-Core.

We should use database instrumentation / logical replicas to stream changes to slave database. In case these database native mechanisms are not available : 
- The easiest transport becomes incremental CSV dumps of tables to S3 or similar cloud storage and restoring on the federated slave database.
- In order to capture incremental changes correctly we need an `updated_at` column per table to filter by and create these CSV's . 
- Deletions though not allowed by Datacube , can happen with direct access to DB and will not be captured and will require a full dump and restore as we are doing now (heavy takes a whole day CPU limited rather than IO limited in Aurora from experiments).
- In post 1.8 we would like to include `updated_at` column in the core table schema and populate this using a trigger installed per table on the [insert and update queries](https://x-team.com/blog/automatic-timestamps-with-postgresql/).


Discussion:

Resolving patch:
diff --git a/datacube/drivers/postgres/_core.py b/datacube/drivers/postgres/_core.py
--- a/datacube/drivers/postgres/_core.py
+++ b/datacube/drivers/postgres/_core.py
@@ -157,7 +157,15 @@ def update_schema(engine: Engine):
     # Empty, as no schema changes have been made recently.
     # -> If you need to write one, look at the Git history of this
     #    function for some examples.
-    return
+    
+    # Post 1.8 DB Federation triggers
+    from datacube.drivers.postgres._triggers import install_timestamp_trigger
+    _LOG.info("Adding Update Triggers")
+    c = engine.connect()
+    c.execute('begin')
+    install_timestamp_trigger(c)
+    c.execute('commit')
+    c.close()
 
 
 def _ensure_role(engine, name, inherits_from=None, add_user=False, create_db=False):
diff --git a/datacube/drivers/postgres/_triggers.py b/datacube/drivers/postgres/_triggers.py
new file mode 100644
--- /dev/null
+++ b/datacube/drivers/postgres/_triggers.py
@@ -0,0 +1,51 @@
+# coding=utf-8
+"""
+Methods for adding triggers to capture row update time-stamps
+"""
+
+from .sql import SCHEMA_NAME
+from ._schema import (
+    METADATA_TYPE, DATASET, DATASET_LOCATION, PRODUCT, DATASET_SOURCE
+)
+
+UPDATE_TIMESTAMP_SQL = """
+create or replace function {schema}.set_row_update_time()
+returns trigger as $$
+begin
+  new.updated = now();
+  return new;
+end;
+$$ language plpgsql;
+""".format(schema=SCHEMA_NAME)
+
+UPDATE_COLUMN_MIGRATE_SQL_TEMPLATE = """
+alter table {schema}.{table} add column if not exists updated
+timestamptz not null default now();
+"""
+
+INSTALL_TRIGGER_SQL_TEMPLATE = """
+drop trigger if exists row_update_time_{table} on {schema}.{table};
+create trigger row_update_time_{table}
+before update on {schema}.{table}
+for each row
+execute procedure {schema}.set_row_update_time();
+"""
+
+TABLE_NAMES = [
+    METADATA_TYPE.name,
+    PRODUCT.name,
+    DATASET_SOURCE.name,
+    DATASET.name,
+    DATASET_LOCATION.name
+]
+
+
+def install_timestamp_trigger(conn):
+    # Create trigger capture function
+    conn.execute(UPDATE_TIMESTAMP_SQL)
+
+    for name in TABLE_NAMES:
+        # Add update_at columns
+        # HACK: Make this more SQLAlchemy with add_column on Table objects
+        conn.execute(UPDATE_COLUMN_MIGRATE_SQL_TEMPLATE.format(schema=SCHEMA_NAME, table=name))
+        conn.execute(INSTALL_TRIGGER_SQL_TEMPLATE.format(schema=SCHEMA_NAME, table=name))


Test patch: