Repository: nearform/temporal_tables
Pull request #88: Include current history migration documentation and optional auto migration option
Author: login: agubler
Created: 2025-03-27T16:34:08Z | Merged: 2025-04-07T08:20:18Z

* Documentation on how to adopt the include current history feature for existing users of temporal tables
* Migration mode that enables gradual migration for records overtime
  * The idea is this will do the hard work for "actively" mutated records over a period of time, at which point the remainder of the records can be manually migrated reducing the risk of missing history

Resolves https://github.com/nearform/temporal_tables/issues/87


Diff:
diff --git a/README.md b/README.md
index 7ea918c..d701a5d 100644
--- a/README.md
+++ b/README.md
@@ -223,7 +223,149 @@ FOR EACH ROW EXECUTE PROCEDURE versioning(
 );
 ```
 
+<a name="migration-to-include-current-version-in-history"></a>
+
+### Migrating to include_current_version_in_history
+
+If you're already using temporal tables and want to adopt the `include_current_version_in_history` feature, follow these steps to safely migrate your existing tables without losing historical data.
+
+#### Prerequisites
+
+1. **Maintenance Window**: Schedule a maintenance window when no applications are writing to the tables.
+2. **Backup**: Create a backup of your database before proceeding.
+3. **Latest Version**: Ensure you're running the latest version of the temporal tables functions:
+   ```sql
+   -- Update the versioning function
+   \i versioning_function.sql
+   ```
+
+#### Migration Steps
+
+1. **Identify Versioned Tables**
+   First, identify all your versioned tables:
+   ```sql
+   SELECT DISTINCT trigger_schema, event_object_table 
+   FROM information_schema.triggers 
+   WHERE trigger_name = 'versioning_trigger'; -- Replace trigger name with the name of the version trigger
+   ```
+
+2. **Copy Current Records**
+   For each versioned table, copy current records to the history table:
+   ```sql
+   -- Replace table_name with your actual table name
+   INSERT INTO table_name_history 
+   SELECT *, tstzrange(LOWER(sys_period), NULL) 
+   FROM table_name 
+   WHERE NOT EXISTS (
+     SELECT 1 
+     FROM table_name_history 
+     WHERE table_name_history.primary_key = table_name.primary_key
+     AND UPPER(table_name_history.sys_period) IS NULL
+   );
+   ```
+
+3. **Update Triggers**
+   Recreate the versioning trigger with the new parameter:
+   ```sql
+   -- Drop existing trigger
+   DROP TRIGGER IF EXISTS versioning_trigger ON table_name;
+   
+   -- Create new trigger with include_current_version_in_history
+   CREATE TRIGGER versioning_trigger
+   BEFORE INSERT OR UPDATE OR DELETE ON table_name
+   FOR EACH ROW EXECUTE PROCEDURE versioning(
+     'sys_period',
+     'table_name_history',
+     true,  -- enforce timestamps
+     false, -- ignore unchanged values (adjust as needed)
+     true   -- include current version in history
+   );
+   ```
+
+#### Verification
+
+After migration, verify the setup:
 
+```sql
+-- Check if current records exist in history
+SELECT t.*, h.*
+FROM table_name t
+LEFT JOIN table_name_history h ON t.primary_key = h.primary_key
+WHERE UPPER(h.sys_period) IS NULL;
+```
+
+#### Rollback Plan
+
+If issues occur, you can revert to the previous state:
+
+```sql
+-- Recreate trigger without include_current_version_in_history
+CREATE OR REPLACE TRIGGER versioning_trigger
+BEFORE INSERT OR UPDATE OR DELETE ON table_name
+FOR EACH ROW EXECUTE PROCEDURE versioning(
+  'sys_period',
+  'table_name_history',
+  true,  -- enforce timestamps
+  false  -- ignore unchanged values (adjust as needed)
+);
+
+-- Remove current versions from history
+DELETE FROM table_name_history 
+WHERE UPPER(sys_period) IS NULL;
+```
+
+#### Notes
+- The migration process ensures no historical data is lost
+- Existing queries that use UNION to combine current and historical records will continue to work
+- New queries can benefit from simplified syntax by querying just the history table
+- Consider updating application queries to use the new consolidated history view
+
+### Automatic Migration Mode
+
+When adopting the `include_current_version_in_history` feature for existing tables, you can use the automatic gradual migration mode to seamlessly populate the history table with current records.
+
+The migration mode is enabled by adding a sixth parameter to the versioning trigger:
+
+```sql
+CREATE TRIGGER versioning_trigger
+BEFORE INSERT OR UPDATE OR DELETE ON your_table
+FOR EACH ROW EXECUTE PROCEDURE versioning(
+  'sys_period', 'your_table_history', true, false, true, true
+);
+```
+
+When migration mode is enabled:
+- On the first `UPDATE` or `DELETE` operation for each record, the trigger checks if the current version exists in the history table
+- If the current version is missing, it's automatically inserted into the history table before proceeding with the update/delete
+- This ensures a complete history is maintained without requiring manual data migration
+
+#### Usage Guidelines
+
+1. **When to Use**: Enable migration mode when you want to adopt `include_current_version_in_history` for existing tables that already have data.
+
+2. **How to Use**:
+   ```sql
+   -- Update your existing trigger to include migration mode
+   DROP TRIGGER IF EXISTS versioning_trigger ON your_table;
+   CREATE TRIGGER versioning_trigger
+   BEFORE INSERT OR UPDATE OR DELETE ON your_table
+   FOR EACH ROW EXECUTE PROCEDURE versioning(
+     'sys_period', 'your_table_history', true, false, true, true
+   );
+   ```
+
+3. **Limitations**:
+   - Migration mode only works for `UPDATE` and `DELETE` operations
+   - It only migrates records that are actually modified or deleted
+   - Once a record has been migrated, subsequent operations will follow normal versioning behavior
+
+4. **Best Practices**:
+   - Enable migration mode only when you're ready to adopt `include_current_version_in_history`
+   - Consider running a test migration on a copy of your data first to determine any performance impacts
+   - Monitor the size of your history table during migration
+   - You can disable migration mode after all records have been migrated
+
+**Note:** The automatic migration happens gradually, filling in missing history only when existing records are updated or deleted. As a result, records that rarely change will still require manual migration using the [method described above](#migration-to-include-current-version-in-history). However, since the most active records will be automatically migrated, the risk of missing important data is greatly reduced, eliminating the need for a dedicated maintenance window.
 
 <a name="migrations"></a>
 
@@ -251,7 +393,19 @@ If the column doesn't accept null values you'll need to modify it to allow for n
 
 ## Test
 
-In order to run tests:
+Ensure you have a postgres database available. A database container can be started by running:
+
+```sh
+npm run db:start
+```
+
+In order to run tests
+
+```sh
+npm test
+```
+
+or
 
 ```sh
 make run_test
@@ -267,6 +421,15 @@ make run_test_nochecks
 
 Obviously, this suite won't run the tests about the error reporting.
 
+**Note:** When running the tests using `make` ensure that the expected environment variables are available
+
+```sh
+PGHOST=localhost
+PGPORT=5432
+PGUSER=postgres
+PGPASSWORD=password
+```
+
 <a name="performance_tests"></a>
 
 ## Performance tests
diff --git a/docker-compose.yml b/docker-compose.yml
new file mode 100644
index 0000000..473ceb6
--- /dev/null
+++ b/docker-compose.yml
@@ -0,0 +1,13 @@
+version: '3.9'
+
+services:
+
+  db:
+    image: postgres:latest
+    container_name: temporal-tables-test
+    restart: always
+    environment:
+      POSTGRES_PASSWORD: password
+      POSTGRES_USER: postgres
+    ports:  
+      - 5432:5432
\ No newline at end of file
diff --git a/package.json b/package.json
index a692ecc..883eb09 100644
--- a/package.json
+++ b/package.json
@@ -7,7 +7,10 @@
     "url": "git+https://github.com/nearform/temporal_tables.git"
   },
   "scripts": {
-    "update-version": "node ./scripts/update-version.js"
+    "db:start": "docker compose up -d",
+    "db:stop": "docker compose down",
+    "update-version": "node ./scripts/update-version.js",
+    "test": "PGHOST=localhost PGPORT=5432 PGUSER=postgres PGPASSWORD=password make run_test"
   },
   "keywords": [],
   "author": "",
diff --git a/test/expected/migration_mode.out b/test/expected/migration_mode.out
new file mode 100644
index 0000000..2197a69
--- /dev/null
+++ b/test/expected/migration_mode.out
@@ -0,0 +1,136 @@
+CREATE EXTENSION IF NOT EXISTS pgcrypto;
+-- Create the table with various columns
+CREATE TABLE versioned_table (
+    id SERIAL PRIMARY KEY,
+    createdAt TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    updatedAt TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    name TEXT,
+    price NUMERIC(10, 2),
+    is_active BOOLEAN DEFAULT true,
+    event_date DATE,
+    metadata JSONB,
+    unique_key UUID DEFAULT gen_random_uuid(),
+    sys_period tstzrange NOT NULL DEFAULT tstzrange(current_timestamp, null)
+);
+-- create history table
+CREATE TABLE versioned_table_history (LIKE versioned_table);
+-- create trigger for versioning without include current and migration
+CREATE TRIGGER versioned_table_versioning BEFORE
+INSERT
+    OR
+UPDATE
+    OR DELETE ON versioned_table FOR EACH ROW EXECUTE PROCEDURE versioning(
+        'sys_period',
+        'versioned_table_history',
+        true
+    );
+--insert 10 records into the table
+INSERT INTO
+    versioned_table (
+        createdAt,
+        updatedAt,
+        name,
+        price,
+        is_active,
+        event_date,
+        metadata
+    )
+SELECT
+    CURRENT_TIMESTAMP,
+    CURRENT_TIMESTAMP,
+    'Item ' || gs,
+    round((random() * 100) :: numeric, 2),
+    (random() < 0.5),
+    CURRENT_DATE + (gs % 30),
+    jsonb_build_object(
+        'generated_id',
+        gs,
+        'random_val',
+        round((random() * 100) :: numeric, 2)
+    )
+FROM
+    generate_series(1, 10) AS gs;
+-- should return count of 0
+SELECT
+    count(*)
+FROM
+    versioned_table_history;
+ count 
+-------
+     0
+(1 row)
+
+-- update a column for all rows
+UPDATE
+    versioned_table
+SET
+    price = round((random() * 100) :: numeric, 2);
+-- should return count of 10
+SELECT
+    count(*)
+FROM
+    versioned_table_history;
+ count 
+-------
+    10
+(1 row)
+
+-- drop versioning trigger
+DROP TRIGGER IF EXISTS versioned_table_versioning ON versioned_table;
+-- Create trigger with auto_migrate and include_current_version_in_history enabled
+CREATE TRIGGER versioned_table_versioning BEFORE
+INSERT
+    OR
+UPDATE
+    OR DELETE ON versioned_table FOR EACH ROW EXECUTE PROCEDURE versioning(
+        'sys_period',
+        'versioned_table_history',
+        true,
+        false,
+        true,
+        true
+    );
+-- update a column for all rows
+UPDATE
+    versioned_table
+SET
+    price = round((random() * 100) :: numeric, 2);
+-- should return count of 30 as the missing history and the current record should now be in the history table
+SELECT
+    count(*)
+FROM
+    versioned_table_history;
+ count 
+-------
+    30
+(1 row)
+
+-- should not return any records
+SELECT
+    count(*),
+    id
+FROM
+    versioned_table_history
+GROUP BY
+    id
+HAVING
+    count(*) != 3;
+ count | id 
+-------+----
+(0 rows)
+
+-- should return count of 10 for the current record of all 10 rows
+SELECT
+    count(*)
+FROM
+    versioned_table_history
+WHERE
+    upper(sys_period) IS NULL;
+ count 
+-------
+    10
+(1 row)
+
+DROP TABLE IF EXISTS versioned_table;
+DROP TABLE IF EXISTS versioned_table_history;
+DROP EXTENSION IF EXISTS pgcrypto;
diff --git a/test/runTest.sh b/test/runTest.sh
index ca4e574..d088da8 100644
--- a/test/runTest.sh
+++ b/test/runTest.sh
@@ -30,6 +30,7 @@ TESTS="
   non_equality_types non_equality_types_unchanged_values
   set_system_time versioning_including_current_version_in_history
   versioning_rollback_include_current_version_in_history noop_update
+  migration_mode
   "
 
 for name in $TESTS; do
diff --git a/test/runTestNochecks.sh b/test/runTestNochecks.sh
index b155461..e87eba6 100644
--- a/test/runTestNochecks.sh
+++ b/test/runTestNochecks.sh
@@ -25,6 +25,7 @@ TESTS="
   non_equality_types non_equality_types_unchanged_values
   unchanged_version_values versioning_including_current_version_in_history
   versioning_rollback_include_current_version_in_history noop_update
+  migration_mode
   "
 
 for name in $TESTS; do
diff --git a/test/sql/migration_mode.sql b/test/sql/migration_mode.sql
new file mode 100644
index 0000000..2cc83e6
--- /dev/null
+++ b/test/sql/migration_mode.sql
@@ -0,0 +1,128 @@
+CREATE EXTENSION IF NOT EXISTS pgcrypto;
+
+-- Create the table with various columns
+CREATE TABLE versioned_table (
+    id SERIAL PRIMARY KEY,
+    createdAt TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    updatedAt TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    name TEXT,
+    price NUMERIC(10, 2),
+    is_active BOOLEAN DEFAULT true,
+    event_date DATE,
+    metadata JSONB,
+    unique_key UUID DEFAULT gen_random_uuid(),
+    sys_period tstzrange NOT NULL DEFAULT tstzrange(current_timestamp, null)
+);
+
+-- create history table
+CREATE TABLE versioned_table_history (LIKE versioned_table);
+
+-- create trigger for versioning without include current and migration
+CREATE TRIGGER versioned_table_versioning BEFORE
+INSERT
+    OR
+UPDATE
+    OR DELETE ON versioned_table FOR EACH ROW EXECUTE PROCEDURE versioning(
+        'sys_period',
+        'versioned_table_history',
+        true
+    );
+
+--insert 10 records into the table
+INSERT INTO
+    versioned_table (
+        createdAt,
+        updatedAt,
+        name,
+        price,
+        is_active,
+        event_date,
+        metadata
+    )
+SELECT
+    CURRENT_TIMESTAMP,
+    CURRENT_TIMESTAMP,
+    'Item ' || gs,
+    round((random() * 100) :: numeric, 2),
+    (random() < 0.5),
+    CURRENT_DATE + (gs % 30),
+    jsonb_build_object(
+        'generated_id',
+        gs,
+        'random_val',
+        round((random() * 100) :: numeric, 2)
+    )
+FROM
+    generate_series(1, 10) AS gs;
+
+-- should return count of 0
+SELECT
+    count(*)
+FROM
+    versioned_table_history;
+
+-- update a column for all rows
+UPDATE
+    versioned_table
+SET
+    price = round((random() * 100) :: numeric, 2);
+
+-- should return count of 10
+SELECT
+    count(*)
+FROM
+    versioned_table_history;
+
+-- drop versioning trigger
+DROP TRIGGER IF EXISTS versioned_table_versioning ON versioned_table;
+
+-- Create trigger with auto_migrate and include_current_version_in_history enabled
+CREATE TRIGGER versioned_table_versioning BEFORE
+INSERT
+    OR
+UPDATE
+    OR DELETE ON versioned_table FOR EACH ROW EXECUTE PROCEDURE versioning(
+        'sys_period',
+        'versioned_table_history',
+        true,
+        false,
+        true,
+        true
+    );
+
+-- update a column for all rows
+UPDATE
+    versioned_table
+SET
+    price = round((random() * 100) :: numeric, 2);
+
+-- should return count of 30 as the missing history and the current record should now be in the history table
+SELECT
+    count(*)
+FROM
+    versioned_table_history;
+
+-- should not return any records
+SELECT
+    count(*),
+    id
+FROM
+    versioned_table_history
+GROUP BY
+    id
+HAVING
+    count(*) != 3;
+
+-- should return count of 10 for the current record of all 10 rows
+SELECT
+    count(*)
+FROM
+    versioned_table_history
+WHERE
+    upper(sys_period) IS NULL;
+
+DROP TABLE IF EXISTS versioned_table;
+
+DROP TABLE IF EXISTS versioned_table_history;
+
+DROP EXTENSION IF EXISTS pgcrypto;
diff --git a/versioning_function.sql b/versioning_function.sql
index e4229cb..90690c2 100644
--- a/versioning_function.sql
+++ b/versioning_function.sql
@@ -9,6 +9,7 @@ DECLARE
   mitigate_update_conflicts text;
   ignore_unchanged_values bool;
   include_current_version_in_history bool;
+  enable_migration_mode bool;
   commonColumns text[];
   time_stamp_to_use timestamptz;
   range_lower timestamptz;
@@ -19,6 +20,7 @@ DECLARE
   newVersion record;
   oldVersion record;
   user_defined_system_time text;
+  record_exists bool;
 BEGIN
   -- set custom system time if exists
   BEGIN
@@ -45,10 +47,10 @@ BEGIN
     MESSAGE = 'function "versioning" must be fired for INSERT or UPDATE or DELETE';
   END IF;
 
-  IF TG_NARGS not between 3 and 5 THEN
+  IF TG_NARGS not between 3 and 6 THEN
     RAISE INVALID_PARAMETER_VALUE USING
     MESSAGE = 'wrong number of parameters for function "versioning"',
-    HINT = 'expected 3 to 5 parameters but got ' || TG_NARGS;
+    HINT = 'expected 3 to 6 parameters but got ' || TG_NARGS;
   END IF;
 
   sys_period := TG_ARGV[0];
@@ -56,6 +58,7 @@ BEGIN
   mitigate_update_conflicts := TG_ARGV[2];
   ignore_unchanged_values := COALESCE(TG_ARGV[3],'false');
   include_current_version_in_history := COALESCE(TG_ARGV[4],'false');
+  enable_migration_mode := COALESCE(TG_ARGV[5],'false');
 
   IF ignore_unchanged_values AND TG_OP = 'UPDATE' THEN
     IF NEW IS NOT DISTINCT FROM OLD THEN
@@ -194,6 +197,32 @@ BEGIN
       INNER JOIN main
       ON history.attname = main.attname
       AND history.attname != sys_period;
+
+    -- Check if record exists in history table for migration mode
+    IF enable_migration_mode = 'true' AND include_current_version_in_history = 'true' AND (TG_OP = 'UPDATE' OR TG_OP = 'DELETE') THEN
+      EXECUTE 'SELECT EXISTS (
+          SELECT 1 FROM ' || history_table || ' WHERE ROW(' ||
+          array_to_string(commonColumns, ',') ||
+          ') IS NOT DISTINCT FROM ROW($1.' ||
+          array_to_string(commonColumns, ',$1.') ||
+          '))'
+      USING OLD INTO record_exists;
+
+      IF NOT record_exists THEN
+        -- Insert current record into history table with its original range
+        EXECUTE 'INSERT INTO ' ||
+          history_table ||
+          '(' ||
+          array_to_string(commonColumns, ',') ||
+          ',' ||
+          quote_ident(sys_period) ||
+          ') VALUES ($1.' ||
+          array_to_string(commonColumns, ',$1.') ||
+          ',tstzrange($2, $3, ''[)''))'
+        USING OLD, range_lower, time_stamp_to_use;
+      END IF;
+    END IF;
+
     -- skip version if it would be identical to the previous version
     IF ignore_unchanged_values AND TG_OP = 'UPDATE' AND array_length(commonColumns, 1) > 0 THEN
       EXECUTE 'SELECT ROW($1.' || array_to_string(commonColumns , ', $1.') || ')'
@@ -206,6 +235,7 @@ BEGIN
         RETURN NEW;
       END IF;
     END IF;
+
     -- If we are including the current version in the history and the operation is an update or delete, we need to update the previous version in the history table
     IF include_current_version_in_history = 'true' THEN
       IF TG_OP = 'UPDATE' OR TG_OP = 'DELETE' THEN
@@ -251,6 +281,7 @@ BEGIN
       ',tstzrange($2, $3, ''[)''))')
        USING OLD, range_lower, time_stamp_to_use;
     END IF;
+
   END IF;
 
   IF TG_OP = 'UPDATE' OR TG_OP = 'INSERT' THEN
diff --git a/versioning_function_nochecks.sql b/versioning_function_nochecks.sql
index 0ddba3c..12c3136 100644
--- a/versioning_function_nochecks.sql
+++ b/versioning_function_nochecks.sql
@@ -9,6 +9,7 @@ DECLARE
   mitigate_update_conflicts text;
   ignore_unchanged_values bool;
   include_current_version_in_history bool;
+  enable_migration_mode bool;
   commonColumns text[];
   time_stamp_to_use timestamptz;
   range_lower timestamptz;
@@ -16,6 +17,7 @@ DECLARE
   newVersion record;
   oldVersion record;
   user_defined_system_time text;
+  record_exists bool;
 BEGIN
   -- set custom system time if exists
   BEGIN
@@ -37,6 +39,7 @@ BEGIN
   mitigate_update_conflicts := TG_ARGV[2];
   ignore_unchanged_values := COALESCE(TG_ARGV[3],'false');
   include_current_version_in_history := COALESCE(TG_ARGV[4],'false');
+  enable_migration_mode := COALESCE(TG_ARGV[5],'false');
 
   IF ignore_unchanged_values AND TG_OP = 'UPDATE' THEN
     IF NEW IS NOT DISTINCT FROM OLD THEN
@@ -87,8 +90,35 @@ BEGIN
       INNER JOIN main
       ON history.attname = main.attname
       AND history.attname != sys_period;
+
+    -- Check if record exists in history table for migration mode
+    IF enable_migration_mode = 'true' AND include_current_version_in_history = 'true' AND (TG_OP = 'UPDATE' OR TG_OP = 'DELETE') THEN
+      EXECUTE 'SELECT EXISTS (
+          SELECT 1 FROM ' || history_table || ' WHERE ROW(' ||
+          array_to_string(commonColumns, ',') ||
+          ') IS NOT DISTINCT FROM ROW($1.' ||
+          array_to_string(commonColumns, ',$1.') ||
+          '))'
+      USING OLD INTO record_exists;
+
+      IF NOT record_exists THEN
+        -- Insert current record into history table with its original range
+        EXECUTE 'INSERT INTO ' ||
+          history_table ||
+          '(' ||
+          array_to_string(commonColumns, ',') ||
+          ',' ||
+          quote_ident(sys_period) ||
+          ') VALUES ($1.' ||
+          array_to_string(commonColumns, ',$1.') ||
+          ',tstzrange($2, $3, ''[)''))'
+        USING OLD, range_lower, time_stamp_to_use;
+      END IF;
+    END IF;
+
     -- skip version if it would be identical to the previous version
-    IF ignore_unchanged_values AND TG_OP = 'UPDATE' AND array_length(commonColumns, 1) > 0 THEN      EXECUTE 'SELECT ROW($1.' || array_to_string(commonColumns , ', $1.') || ')'
+    IF ignore_unchanged_values AND TG_OP = 'UPDATE' AND array_length(commonColumns, 1) > 0 THEN
+      EXECUTE 'SELECT ROW($1.' || array_to_string(commonColumns , ', $1.') || ')'
         USING NEW
         INTO newVersion;
       EXECUTE 'SELECT ROW($1.' || array_to_string(commonColumns , ', $1.') || ')'
@@ -98,7 +128,8 @@ BEGIN
         RETURN NEW;
       END IF;
     END IF;
--- If we are including the current version in the history and the operation is an update or delete, we need to update the previous version in the history table
+
+    -- If we are including the current version in the history and the operation is an update or delete, we need to update the previous version in the history table
     IF include_current_version_in_history = 'true' THEN
       IF TG_OP = 'UPDATE' OR TG_OP = 'DELETE' THEN
         EXECUTE (


Reviews:
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR adds documentation for migrating existing temporal tables to include the current version in history and introduces an optional automatic gradual migration mode.  
- Added detailed migration steps and prerequisites for safely copying current records into history tables.  
- Introduced an optional trigger configuration for automatic migration mode with examples.

<details>
<summary>Files not reviewed (7)</summary>

* **package.json**: Language not supported
* **test/expected/migration_mode.out**: Language not supported
* **test/runTest.sh**: Language not supported
* **test/runTestNochecks.sh**: Language not supported
* **test/sql/migration_mode.sql**: Language not supported
* **versioning_function.sql**: Language not supported
* **versioning_function_nochecks.sql**: Language not supported
</details>



state: COMMENTED
author_association: NONE
submitted_at: 2025-04-02T07:31:26Z
user: login: simoneb
body: Great job here 🚀 I've added a few comments to help me establish some context for the review, before going deeper into it.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-02T07:40:03Z
user: login: agubler
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T15:25:30Z
user: login: agubler
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T15:25:32Z
user: login: agubler
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T15:26:00Z
user: login: simoneb
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-03T09:57:52Z
user: login: simoneb
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-03T10:00:07Z
user: login: simoneb
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-03T10:01:35Z
user: login: agubler
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T07:42:10Z
user: login: simoneb
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T07:55:10Z
user: login: agubler
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T15:52:33Z
user: login: simoneb
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T15:53:45Z
user: login: simoneb
body: please undo this last change, it's done automatically by the release process
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T07:50:42Z

Review comments:
diff_hunk: @@ -223,7 +223,149 @@ FOR EACH ROW EXECUTE PROCEDURE versioning(
 );
 ```
 
+<a name="migration-to-include-current-version-in-history"></a>
+
+### Migrating to include_current_version_in_history
+
+If you're already using temporal tables and want to adopt the `include_current_version_in_history` feature, follow these steps to safely migrate your existing tables without losing historical data.
+
+#### Prerequisites
+
+1. **Maintenance Window**: Schedule a maintenance window when no applications are writing to the tables.
+2. **Backup**: Create a backup of your database before proceeding.
+3. **Latest Version**: Ensure you're running the latest version of the temporal tables functions:
+   ```sql
+   -- Update the versioning function
+   \i versioning_function.sql
+   ```
+
+#### Migration Steps
+
+1. **Identify Versioned Tables**
+   First, identify all your versioned tables:
+   ```sql
+   SELECT DISTINCT trigger_schema, event_object_table 
+   FROM information_schema.triggers 
+   WHERE trigger_name = 'versioning_trigger'; -- Replace trigger name with the name of the version trigger
+   ```
+
+2. **Copy Current Records**
+   For each versioned table, copy current records to the history table:
+   ```sql
+   -- Replace table_name with your actual table name
+   INSERT INTO table_name_history 
+   SELECT *, tstzrange(LOWER(sys_period), NULL) 
path: README.md
user: login: Copilot
body: [nitpick] Using SELECT * in this migration snippet may lead to issues if the schema changes; consider explicitly specifying the column names to ensure the correct mapping between the source and history tables.
```suggestion
   INSERT INTO table_name_history (column1, column2, column3, sys_period)
   SELECT column1, column2, column3, tstzrange(LOWER(sys_period), NULL) 
```
created_at: 2025-04-02T07:31:26Z
author_association: NONE
diff_hunk: @@ -223,7 +223,149 @@ FOR EACH ROW EXECUTE PROCEDURE versioning(
 );
 ```
 
+<a name="migration-to-include-current-version-in-history"></a>
+
+### Migrating to include_current_version_in_history
+
+If you're already using temporal tables and want to adopt the `include_current_version_in_history` feature, follow these steps to safely migrate your existing tables without losing historical data.
+
+#### Prerequisites
+
+1. **Maintenance Window**: Schedule a maintenance window when no applications are writing to the tables.
+2. **Backup**: Create a backup of your database before proceeding.
+3. **Latest Version**: Ensure you're running the latest version of the temporal tables functions:
+   ```sql
+   -- Update the versioning function
+   \i versioning_function.sql
+   ```
+
+#### Migration Steps
+
+1. **Identify Versioned Tables**
+   First, identify all your versioned tables:
+   ```sql
+   SELECT DISTINCT trigger_schema, event_object_table 
+   FROM information_schema.triggers 
+   WHERE trigger_name = 'versioning_trigger'; -- Replace trigger name with the name of the version trigger
+   ```
+
+2. **Copy Current Records**
+   For each versioned table, copy current records to the history table:
+   ```sql
+   -- Replace table_name with your actual table name
+   INSERT INTO table_name_history 
+   SELECT *, tstzrange(LOWER(sys_period), NULL) 
+   FROM table_name 
+   WHERE NOT EXISTS (
+     SELECT 1 
+     FROM table_name_history 
+     WHERE table_name_history.primary_key = table_name.primary_key
+     AND UPPER(table_name_history.sys_period) IS NULL
+   );
+   ```
+
+3. **Update Triggers**
+   Recreate the versioning trigger with the new parameter:
+   ```sql
+   -- Drop existing trigger
+   DROP TRIGGER IF EXISTS versioning_trigger ON table_name;
+   
+   -- Create new trigger with include_current_version_in_history
+   CREATE TRIGGER versioning_trigger
+   BEFORE INSERT OR UPDATE OR DELETE ON table_name
+   FOR EACH ROW EXECUTE PROCEDURE versioning(
+     'sys_period',
+     'table_name_history',
+     true,  -- enforce timestamps
+     false, -- ignore unchanged values (adjust as needed)
+     true   -- include current version in history
+   );
+   ```
+
+#### Verification
+
+After migration, verify the setup:
 
+```sql
+-- Check if current records exist in history
+SELECT t.*, h.*
+FROM table_name t
+LEFT JOIN table_name_history h ON t.primary_key = h.primary_key
+WHERE UPPER(h.sys_period) IS NULL;
+```
+
+#### Rollback Plan
+
+If issues occur, you can revert to the previous state:
+
+```sql
+-- Recreate trigger without include_current_version_in_history
+CREATE OR REPLACE TRIGGER versioning_trigger
+BEFORE INSERT OR UPDATE OR DELETE ON table_name
+FOR EACH ROW EXECUTE PROCEDURE versioning(
+  'sys_period',
+  'table_name_history',
+  true,  -- enforce timestamps
+  false  -- ignore unchanged values (adjust as needed)
+);
+
+-- Remove current versions from history
+DELETE FROM table_name_history 
+WHERE UPPER(sys_period) IS NULL;
+```
+
+#### Notes
+- The migration process ensures no historical data is lost
+- Existing queries that use UNION to combine current and historical records will continue to work
+- New queries can benefit from simplified syntax by querying just the history table
+- Consider updating application queries to use the new consolidated history view
+
+### Automatic Migration Mode
+
+When adopting the `include_current_version_in_history` feature for existing tables, you can use the automatic gradual migration mode to seamlessly populate the history table with current records.
+
+The migration mode is enabled by adding a sixth parameter to the versioning trigger:
+
+```sql
+CREATE TRIGGER versioning_trigger
+BEFORE INSERT OR UPDATE OR DELETE ON your_table
+FOR EACH ROW EXECUTE PROCEDURE versioning(
+  'sys_period', 'your_table_history', true, false, true, true
+);
+```
+
+When migration mode is enabled:
+- On the first `UPDATE` or `DELETE` operation for each record, the trigger checks if the current version exists in the history table
+- If the current version is missing, it's automatically inserted into the history table before proceeding with the update/delete
+- This ensures a complete history is maintained without requiring manual data migration
+
+#### Usage Guidelines
+
+1. **When to Use**: Enable migration mode when you want to adopt `include_current_version_in_history` for existing tables that already have data.
+
+2. **How to Use**:
+   ```sql
+   -- Update your existing trigger to include migration mode
+   DROP TRIGGER IF EXISTS versioning_trigger ON your_table;
+   CREATE TRIGGER versioning_trigger
+   BEFORE INSERT OR UPDATE OR DELETE ON your_table
+   FOR EACH ROW EXECUTE PROCEDURE versioning(
+     'sys_period', 'your_table_history', true, false, true, true
+   );
+   ```
+
+3. **Limitations**:
+   - Migration mode only works for `UPDATE` and `DELETE` operations
+   - It only migrates records that are actually modified or deleted
path: README.md
user: login: simoneb
body: what does this mean? or iow, what does this not cover then?
created_at: 2025-04-02T07:36:13Z
author_association: MEMBER
diff_hunk: @@ -223,7 +223,149 @@ FOR EACH ROW EXECUTE PROCEDURE versioning(
 );
 ```
 
+<a name="migration-to-include-current-version-in-history"></a>
+
+### Migrating to include_current_version_in_history
+
+If you're already using temporal tables and want to adopt the `include_current_version_in_history` feature, follow these steps to safely migrate your existing tables without losing historical data.
+
+#### Prerequisites
+
+1. **Maintenance Window**: Schedule a maintenance window when no applications are writing to the tables.
+2. **Backup**: Create a backup of your database before proceeding.
+3. **Latest Version**: Ensure you're running the latest version of the temporal tables functions:
+   ```sql
+   -- Update the versioning function
+   \i versioning_function.sql
+   ```
+
+#### Migration Steps
+
+1. **Identify Versioned Tables**
+   First, identify all your versioned tables:
+   ```sql
+   SELECT DISTINCT trigger_schema, event_object_table 
+   FROM information_schema.triggers 
+   WHERE trigger_name = 'versioning_trigger'; -- Replace trigger name with the name of the version trigger
+   ```
+
+2. **Copy Current Records**
+   For each versioned table, copy current records to the history table:
+   ```sql
+   -- Replace table_name with your actual table name
+   INSERT INTO table_name_history 
+   SELECT *, tstzrange(LOWER(sys_period), NULL) 
+   FROM table_name 
+   WHERE NOT EXISTS (
+     SELECT 1 
+     FROM table_name_history 
+     WHERE table_name_history.primary_key = table_name.primary_key
+     AND UPPER(table_name_history.sys_period) IS NULL
+   );
+   ```
+
+3. **Update Triggers**
+   Recreate the versioning trigger with the new parameter:
+   ```sql
+   -- Drop existing trigger
+   DROP TRIGGER IF EXISTS versioning_trigger ON table_name;
+   
+   -- Create new trigger with include_current_version_in_history
+   CREATE TRIGGER versioning_trigger
+   BEFORE INSERT OR UPDATE OR DELETE ON table_name
+   FOR EACH ROW EXECUTE PROCEDURE versioning(
+     'sys_period',
+     'table_name_history',
+     true,  -- enforce timestamps
+     false, -- ignore unchanged values (adjust as needed)
+     true   -- include current version in history
+   );
+   ```
+
+#### Verification
+
+After migration, verify the setup:
 
+```sql
+-- Check if current records exist in history
+SELECT t.*, h.*
+FROM table_name t
+LEFT JOIN table_name_history h ON t.primary_key = h.primary_key
+WHERE UPPER(h.sys_period) IS NULL;
+```
+
+#### Rollback Plan
+
+If issues occur, you can revert to the previous state:
+
+```sql
+-- Recreate trigger without include_current_version_in_history
+CREATE OR REPLACE TRIGGER versioning_trigger
+BEFORE INSERT OR UPDATE OR DELETE ON table_name
+FOR EACH ROW EXECUTE PROCEDURE versioning(
+  'sys_period',
+  'table_name_history',
+  true,  -- enforce timestamps
+  false  -- ignore unchanged values (adjust as needed)
+);
+
+-- Remove current versions from history
+DELETE FROM table_name_history 
+WHERE UPPER(sys_period) IS NULL;
+```
+
+#### Notes
+- The migration process ensures no historical data is lost
+- Existing queries that use UNION to combine current and historical records will continue to work
+- New queries can benefit from simplified syntax by querying just the history table
+- Consider updating application queries to use the new consolidated history view
+
+### Automatic Migration Mode
+
+When adopting the `include_current_version_in_history` feature for existing tables, you can use the automatic gradual migration mode to seamlessly populate the history table with current records.
+
+The migration mode is enabled by adding a sixth parameter to the versioning trigger:
+
+```sql
+CREATE TRIGGER versioning_trigger
+BEFORE INSERT OR UPDATE OR DELETE ON your_table
+FOR EACH ROW EXECUTE PROCEDURE versioning(
+  'sys_period', 'your_table_history', true, false, true, true
+);
+```
+
+When migration mode is enabled:
+- On the first `UPDATE` or `DELETE` operation for each record, the trigger checks if the current version exists in the history table
+- If the current version is missing, it's automatically inserted into the history table before proceeding with the update/delete
+- This ensures a complete history is maintained without requiring manual data migration
+
+#### Usage Guidelines
+
+1. **When to Use**: Enable migration mode when you want to adopt `include_current_version_in_history` for existing tables that already have data.
+
+2. **How to Use**:
+   ```sql
+   -- Update your existing trigger to include migration mode
+   DROP TRIGGER IF EXISTS versioning_trigger ON your_table;
+   CREATE TRIGGER versioning_trigger
+   BEFORE INSERT OR UPDATE OR DELETE ON your_table
+   FOR EACH ROW EXECUTE PROCEDURE versioning(
+     'sys_period', 'your_table_history', true, false, true, true
+   );
+   ```
+
+3. **Limitations**:
+   - Migration mode only works for `UPDATE` and `DELETE` operations
+   - It only migrates records that are actually modified or deleted
+   - Once a record has been migrated, subsequent operations will follow normal versioning behavior
+
+4. **Best Practices**:
+   - Enable migration mode only when you're ready to adopt `include_current_version_in_history`
+   - Consider running a test migration on a copy of your data first to determine any performance impacts
+   - Monitor the size of your history table during migration
+   - You can disable migration mode after all records have been migrated
+
+**Note:** The automatic migration happens gradually, filling in missing history only when existing records are updated or deleted. As a result, records that rarely change will still require manual migration using the [method described above](#migration-to-include-current-version-in-history). However, since the most active records will be automatically migrated, the risk of missing important data is greatly reduced, eliminating the need for a dedicated maintenance window.
path: README.md
user: login: simoneb
body: Is it correct to say that "records that rarely change will still..."? Is it not rather about "tables that rarely change"?
created_at: 2025-04-02T07:37:09Z
author_association: MEMBER
diff_hunk: @@ -7,7 +7,11 @@
     "url": "git+https://github.com/nearform/temporal_tables.git"
   },
   "scripts": {
-    "update-version": "node ./scripts/update-version.js"
+    "update-version": "node ./scripts/update-version.js",
+    "pretest": "docker run --name temporal-tables-test -e POSTGRES_PASSWORD=password -e POSTGRES_USER=postgres -p 5432:5432 -d postgres:latest && sleep 3 && docker exec temporal-tables-test pg_isready -h localhost -p 5432 -U postgres -t 30",
+    "test": "(PGHOST=localhost PGPORT=5432 PGUSER=postgres PGPASSWORD=password make run_test; ret=$?; docker stop temporal-tables-test && docker rm temporal-tables-test; exit $ret)",
+    "test:nochecks": "npm run pretest && (PGHOST=localhost PGPORT=5432 PGUSER=postgres PGPASSWORD=password make run_test_nochecks; ret=$?; docker stop temporal-tables-test && docker rm temporal-tables-test; exit $ret)",
+    "cleanup": "docker stop temporal-tables-test || true && docker rm temporal-tables-test || true"
path: package.json
user: login: simoneb
body: why do we need new npm scripts?
created_at: 2025-04-02T07:38:19Z
author_association: MEMBER
diff_hunk: @@ -223,7 +223,149 @@ FOR EACH ROW EXECUTE PROCEDURE versioning(
 );
 ```
 
+<a name="migration-to-include-current-version-in-history"></a>
+
+### Migrating to include_current_version_in_history
+
+If you're already using temporal tables and want to adopt the `include_current_version_in_history` feature, follow these steps to safely migrate your existing tables without losing historical data.
+
+#### Prerequisites
+
+1. **Maintenance Window**: Schedule a maintenance window when no applications are writing to the tables.
+2. **Backup**: Create a backup of your database before proceeding.
+3. **Latest Version**: Ensure you're running the latest version of the temporal tables functions:
+   ```sql
+   -- Update the versioning function
+   \i versioning_function.sql
+   ```
+
+#### Migration Steps
+
+1. **Identify Versioned Tables**
+   First, identify all your versioned tables:
+   ```sql
+   SELECT DISTINCT trigger_schema, event_object_table 
+   FROM information_schema.triggers 
+   WHERE trigger_name = 'versioning_trigger'; -- Replace trigger name with the name of the version trigger
+   ```
+
+2. **Copy Current Records**
+   For each versioned table, copy current records to the history table:
+   ```sql
+   -- Replace table_name with your actual table name
+   INSERT INTO table_name_history 
+   SELECT *, tstzrange(LOWER(sys_period), NULL) 
+   FROM table_name 
+   WHERE NOT EXISTS (
+     SELECT 1 
+     FROM table_name_history 
+     WHERE table_name_history.primary_key = table_name.primary_key
+     AND UPPER(table_name_history.sys_period) IS NULL
+   );
+   ```
+
+3. **Update Triggers**
+   Recreate the versioning trigger with the new parameter:
+   ```sql
+   -- Drop existing trigger
+   DROP TRIGGER IF EXISTS versioning_trigger ON table_name;
+   
+   -- Create new trigger with include_current_version_in_history
+   CREATE TRIGGER versioning_trigger
+   BEFORE INSERT OR UPDATE OR DELETE ON table_name
+   FOR EACH ROW EXECUTE PROCEDURE versioning(
+     'sys_period',
+     'table_name_history',
+     true,  -- enforce timestamps
+     false, -- ignore unchanged values (adjust as needed)
+     true   -- include current version in history
+   );
+   ```
+
+#### Verification
+
+After migration, verify the setup:
 
+```sql
+-- Check if current records exist in history
+SELECT t.*, h.*
+FROM table_name t
+LEFT JOIN table_name_history h ON t.primary_key = h.primary_key
+WHERE UPPER(h.sys_period) IS NULL;
+```
+
+#### Rollback Plan
+
+If issues occur, you can revert to the previous state:
+
+```sql
+-- Recreate trigger without include_current_version_in_history
+CREATE OR REPLACE TRIGGER versioning_trigger
+BEFORE INSERT OR UPDATE OR DELETE ON table_name
+FOR EACH ROW EXECUTE PROCEDURE versioning(
+  'sys_period',
+  'table_name_history',
+  true,  -- enforce timestamps
+  false  -- ignore unchanged values (adjust as needed)
+);
+
+-- Remove current versions from history
+DELETE FROM table_name_history 
+WHERE UPPER(sys_period) IS NULL;
+```
+
+#### Notes
+- The migration process ensures no historical data is lost
+- Existing queries that use UNION to combine current and historical records will continue to work
+- New queries can benefit from simplified syntax by querying just the history table
+- Consider updating application queries to use the new consolidated history view
+
+### Automatic Migration Mode
+
+When adopting the `include_current_version_in_history` feature for existing tables, you can use the automatic gradual migration mode to seamlessly populate the history table with current records.
+
+The migration mode is enabled by adding a sixth parameter to the versioning trigger:
+
+```sql
+CREATE TRIGGER versioning_trigger
+BEFORE INSERT OR UPDATE OR DELETE ON your_table
+FOR EACH ROW EXECUTE PROCEDURE versioning(
+  'sys_period', 'your_table_history', true, false, true, true
+);
+```
+
+When migration mode is enabled:
+- On the first `UPDATE` or `DELETE` operation for each record, the trigger checks if the current version exists in the history table
+- If the current version is missing, it's automatically inserted into the history table before proceeding with the update/delete
+- This ensures a complete history is maintained without requiring manual data migration
+
+#### Usage Guidelines
+
+1. **When to Use**: Enable migration mode when you want to adopt `include_current_version_in_history` for existing tables that already have data.
+
+2. **How to Use**:
+   ```sql
+   -- Update your existing trigger to include migration mode
+   DROP TRIGGER IF EXISTS versioning_trigger ON your_table;
+   CREATE TRIGGER versioning_trigger
+   BEFORE INSERT OR UPDATE OR DELETE ON your_table
+   FOR EACH ROW EXECUTE PROCEDURE versioning(
+     'sys_period', 'your_table_history', true, false, true, true
+   );
+   ```
+
+3. **Limitations**:
+   - Migration mode only works for `UPDATE` and `DELETE` operations
+   - It only migrates records that are actually modified or deleted
path: README.md
user: login: agubler
body: Meaning that within the trigger it does not migrate all the missing records for the table just the record that is being updated or inserted, doing the entire table would not be good for large datasets given it would impact the performance of the insert/update
created_at: 2025-04-02T15:25:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -223,7 +223,149 @@ FOR EACH ROW EXECUTE PROCEDURE versioning(
 );
 ```
 
+<a name="migration-to-include-current-version-in-history"></a>
+
+### Migrating to include_current_version_in_history
+
+If you're already using temporal tables and want to adopt the `include_current_version_in_history` feature, follow these steps to safely migrate your existing tables without losing historical data.
+
+#### Prerequisites
+
+1. **Maintenance Window**: Schedule a maintenance window when no applications are writing to the tables.
+2. **Backup**: Create a backup of your database before proceeding.
+3. **Latest Version**: Ensure you're running the latest version of the temporal tables functions:
+   ```sql
+   -- Update the versioning function
+   \i versioning_function.sql
+   ```
+
+#### Migration Steps
+
+1. **Identify Versioned Tables**
+   First, identify all your versioned tables:
+   ```sql
+   SELECT DISTINCT trigger_schema, event_object_table 
+   FROM information_schema.triggers 
+   WHERE trigger_name = 'versioning_trigger'; -- Replace trigger name with the name of the version trigger
+   ```
+
+2. **Copy Current Records**
+   For each versioned table, copy current records to the history table:
+   ```sql
+   -- Replace table_name with your actual table name
+   INSERT INTO table_name_history 
+   SELECT *, tstzrange(LOWER(sys_period), NULL) 
+   FROM table_name 
+   WHERE NOT EXISTS (
+     SELECT 1 
+     FROM table_name_history 
+     WHERE table_name_history.primary_key = table_name.primary_key
+     AND UPPER(table_name_history.sys_period) IS NULL
+   );
+   ```
+
+3. **Update Triggers**
+   Recreate the versioning trigger with the new parameter:
+   ```sql
+   -- Drop existing trigger
+   DROP TRIGGER IF EXISTS versioning_trigger ON table_name;
+   
+   -- Create new trigger with include_current_version_in_history
+   CREATE TRIGGER versioning_trigger
+   BEFORE INSERT OR UPDATE OR DELETE ON table_name
+   FOR EACH ROW EXECUTE PROCEDURE versioning(
+     'sys_period',
+     'table_name_history',
+     true,  -- enforce timestamps
+     false, -- ignore unchanged values (adjust as needed)
+     true   -- include current version in history
+   );
+   ```
+
+#### Verification
+
+After migration, verify the setup:
 
+```sql
+-- Check if current records exist in history
+SELECT t.*, h.*
+FROM table_name t
+LEFT JOIN table_name_history h ON t.primary_key = h.primary_key
+WHERE UPPER(h.sys_period) IS NULL;
+```
+
+#### Rollback Plan
+
+If issues occur, you can revert to the previous state:
+
+```sql
+-- Recreate trigger without include_current_version_in_history
+CREATE OR REPLACE TRIGGER versioning_trigger
+BEFORE INSERT OR UPDATE OR DELETE ON table_name
+FOR EACH ROW EXECUTE PROCEDURE versioning(
+  'sys_period',
+  'table_name_history',
+  true,  -- enforce timestamps
+  false  -- ignore unchanged values (adjust as needed)
+);
+
+-- Remove current versions from history
+DELETE FROM table_name_history 
+WHERE UPPER(sys_period) IS NULL;
+```
+
+#### Notes
+- The migration process ensures no historical data is lost
+- Existing queries that use UNION to combine current and historical records will continue to work
+- New queries can benefit from simplified syntax by querying just the history table
+- Consider updating application queries to use the new consolidated history view
+
+### Automatic Migration Mode
+
+When adopting the `include_current_version_in_history` feature for existing tables, you can use the automatic gradual migration mode to seamlessly populate the history table with current records.
+
+The migration mode is enabled by adding a sixth parameter to the versioning trigger:
+
+```sql
+CREATE TRIGGER versioning_trigger
+BEFORE INSERT OR UPDATE OR DELETE ON your_table
+FOR EACH ROW EXECUTE PROCEDURE versioning(
+  'sys_period', 'your_table_history', true, false, true, true
+);
+```
+
+When migration mode is enabled:
+- On the first `UPDATE` or `DELETE` operation for each record, the trigger checks if the current version exists in the history table
+- If the current version is missing, it's automatically inserted into the history table before proceeding with the update/delete
+- This ensures a complete history is maintained without requiring manual data migration
+
+#### Usage Guidelines
+
+1. **When to Use**: Enable migration mode when you want to adopt `include_current_version_in_history` for existing tables that already have data.
+
+2. **How to Use**:
+   ```sql
+   -- Update your existing trigger to include migration mode
+   DROP TRIGGER IF EXISTS versioning_trigger ON your_table;
+   CREATE TRIGGER versioning_trigger
+   BEFORE INSERT OR UPDATE OR DELETE ON your_table
+   FOR EACH ROW EXECUTE PROCEDURE versioning(
+     'sys_period', 'your_table_history', true, false, true, true
+   );
+   ```
+
+3. **Limitations**:
+   - Migration mode only works for `UPDATE` and `DELETE` operations
+   - It only migrates records that are actually modified or deleted
+   - Once a record has been migrated, subsequent operations will follow normal versioning behavior
+
+4. **Best Practices**:
+   - Enable migration mode only when you're ready to adopt `include_current_version_in_history`
+   - Consider running a test migration on a copy of your data first to determine any performance impacts
+   - Monitor the size of your history table during migration
+   - You can disable migration mode after all records have been migrated
+
+**Note:** The automatic migration happens gradually, filling in missing history only when existing records are updated or deleted. As a result, records that rarely change will still require manual migration using the [method described above](#migration-to-include-current-version-in-history). However, since the most active records will be automatically migrated, the risk of missing important data is greatly reduced, eliminating the need for a dedicated maintenance window.
path: README.md
user: login: agubler
body: Well that's part of the limitation, it's a gradual record by record migration so it will assist by adding the "missing" history record as records are modified but doesn't guarantee the entire table will be migrated if a record in the main table is not updated or deleted then the current record will still not be in the history table
created_at: 2025-04-02T15:25:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,7 +7,11 @@
     "url": "git+https://github.com/nearform/temporal_tables.git"
   },
   "scripts": {
-    "update-version": "node ./scripts/update-version.js"
+    "update-version": "node ./scripts/update-version.js",
+    "pretest": "docker run --name temporal-tables-test -e POSTGRES_PASSWORD=password -e POSTGRES_USER=postgres -p 5432:5432 -d postgres:latest && sleep 3 && docker exec temporal-tables-test pg_isready -h localhost -p 5432 -U postgres -t 30",
+    "test": "(PGHOST=localhost PGPORT=5432 PGUSER=postgres PGPASSWORD=password make run_test; ret=$?; docker stop temporal-tables-test && docker rm temporal-tables-test; exit $ret)",
+    "test:nochecks": "npm run pretest && (PGHOST=localhost PGPORT=5432 PGUSER=postgres PGPASSWORD=password make run_test_nochecks; ret=$?; docker stop temporal-tables-test && docker rm temporal-tables-test; exit $ret)",
+    "cleanup": "docker stop temporal-tables-test || true && docker rm temporal-tables-test || true"
path: package.json
user: login: agubler
body: Previously there was only `make run_test` which had an assumption that a postgres db was available to run against - I found it slightly obstructive so added scripts that would start the db with the right env vars that the tests expected and then stopped it on completion - I can remove if we don't want to have this?
created_at: 2025-04-02T15:26:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -223,7 +223,149 @@ FOR EACH ROW EXECUTE PROCEDURE versioning(
 );
 ```
 
+<a name="migration-to-include-current-version-in-history"></a>
+
+### Migrating to include_current_version_in_history
+
+If you're already using temporal tables and want to adopt the `include_current_version_in_history` feature, follow these steps to safely migrate your existing tables without losing historical data.
+
+#### Prerequisites
+
+1. **Maintenance Window**: Schedule a maintenance window when no applications are writing to the tables.
+2. **Backup**: Create a backup of your database before proceeding.
+3. **Latest Version**: Ensure you're running the latest version of the temporal tables functions:
+   ```sql
+   -- Update the versioning function
+   \i versioning_function.sql
+   ```
+
+#### Migration Steps
+
+1. **Identify Versioned Tables**
+   First, identify all your versioned tables:
+   ```sql
+   SELECT DISTINCT trigger_schema, event_object_table 
+   FROM information_schema.triggers 
+   WHERE trigger_name = 'versioning_trigger'; -- Replace trigger name with the name of the version trigger
+   ```
+
+2. **Copy Current Records**
+   For each versioned table, copy current records to the history table:
+   ```sql
+   -- Replace table_name with your actual table name
+   INSERT INTO table_name_history 
+   SELECT *, tstzrange(LOWER(sys_period), NULL) 
+   FROM table_name 
+   WHERE NOT EXISTS (
+     SELECT 1 
+     FROM table_name_history 
+     WHERE table_name_history.primary_key = table_name.primary_key
+     AND UPPER(table_name_history.sys_period) IS NULL
+   );
+   ```
+
+3. **Update Triggers**
+   Recreate the versioning trigger with the new parameter:
+   ```sql
+   -- Drop existing trigger
+   DROP TRIGGER IF EXISTS versioning_trigger ON table_name;
+   
+   -- Create new trigger with include_current_version_in_history
+   CREATE TRIGGER versioning_trigger
+   BEFORE INSERT OR UPDATE OR DELETE ON table_name
+   FOR EACH ROW EXECUTE PROCEDURE versioning(
+     'sys_period',
+     'table_name_history',
+     true,  -- enforce timestamps
+     false, -- ignore unchanged values (adjust as needed)
+     true   -- include current version in history
+   );
+   ```
+
+#### Verification
+
+After migration, verify the setup:
 
+```sql
+-- Check if current records exist in history
+SELECT t.*, h.*
+FROM table_name t
+LEFT JOIN table_name_history h ON t.primary_key = h.primary_key
+WHERE UPPER(h.sys_period) IS NULL;
+```
+
+#### Rollback Plan
+
+If issues occur, you can revert to the previous state:
+
+```sql
+-- Recreate trigger without include_current_version_in_history
+CREATE OR REPLACE TRIGGER versioning_trigger
+BEFORE INSERT OR UPDATE OR DELETE ON table_name
+FOR EACH ROW EXECUTE PROCEDURE versioning(
+  'sys_period',
+  'table_name_history',
+  true,  -- enforce timestamps
+  false  -- ignore unchanged values (adjust as needed)
+);
+
+-- Remove current versions from history
+DELETE FROM table_name_history 
+WHERE UPPER(sys_period) IS NULL;
+```
+
+#### Notes
+- The migration process ensures no historical data is lost
+- Existing queries that use UNION to combine current and historical records will continue to work
+- New queries can benefit from simplified syntax by querying just the history table
+- Consider updating application queries to use the new consolidated history view
+
+### Automatic Migration Mode
+
+When adopting the `include_current_version_in_history` feature for existing tables, you can use the automatic gradual migration mode to seamlessly populate the history table with current records.
+
+The migration mode is enabled by adding a sixth parameter to the versioning trigger:
+
+```sql
+CREATE TRIGGER versioning_trigger
+BEFORE INSERT OR UPDATE OR DELETE ON your_table
+FOR EACH ROW EXECUTE PROCEDURE versioning(
+  'sys_period', 'your_table_history', true, false, true, true
+);
+```
+
+When migration mode is enabled:
+- On the first `UPDATE` or `DELETE` operation for each record, the trigger checks if the current version exists in the history table
+- If the current version is missing, it's automatically inserted into the history table before proceeding with the update/delete
+- This ensures a complete history is maintained without requiring manual data migration
+
+#### Usage Guidelines
+
+1. **When to Use**: Enable migration mode when you want to adopt `include_current_version_in_history` for existing tables that already have data.
+
+2. **How to Use**:
+   ```sql
+   -- Update your existing trigger to include migration mode
+   DROP TRIGGER IF EXISTS versioning_trigger ON your_table;
+   CREATE TRIGGER versioning_trigger
+   BEFORE INSERT OR UPDATE OR DELETE ON your_table
+   FOR EACH ROW EXECUTE PROCEDURE versioning(
+     'sys_period', 'your_table_history', true, false, true, true
+   );
+   ```
+
+3. **Limitations**:
+   - Migration mode only works for `UPDATE` and `DELETE` operations
+   - It only migrates records that are actually modified or deleted
path: README.md
user: login: simoneb
body: ok yeah that makes sense, and that's really the only sensible thing to do
created_at: 2025-04-03T09:57:52Z
author_association: MEMBER
diff_hunk: @@ -7,7 +7,11 @@
     "url": "git+https://github.com/nearform/temporal_tables.git"
   },
   "scripts": {
-    "update-version": "node ./scripts/update-version.js"
+    "update-version": "node ./scripts/update-version.js",
+    "pretest": "docker run --name temporal-tables-test -e POSTGRES_PASSWORD=password -e POSTGRES_USER=postgres -p 5432:5432 -d postgres:latest && sleep 3 && docker exec temporal-tables-test pg_isready -h localhost -p 5432 -U postgres -t 30",
+    "test": "(PGHOST=localhost PGPORT=5432 PGUSER=postgres PGPASSWORD=password make run_test; ret=$?; docker stop temporal-tables-test && docker rm temporal-tables-test; exit $ret)",
+    "test:nochecks": "npm run pretest && (PGHOST=localhost PGPORT=5432 PGUSER=postgres PGPASSWORD=password make run_test_nochecks; ret=$?; docker stop temporal-tables-test && docker rm temporal-tables-test; exit $ret)",
+    "cleanup": "docker stop temporal-tables-test || true && docker rm temporal-tables-test || true"
path: package.json
user: login: simoneb
body: I find it a bit overkill quite frankly, and especially obscure. since you did the work, I'm not going to suggest that you remove it, but I think that just having a dockerfile in the repo would have achieved 80% with 20% of the effort and complication
created_at: 2025-04-03T10:00:07Z
author_association: MEMBER
diff_hunk: @@ -7,7 +7,11 @@
     "url": "git+https://github.com/nearform/temporal_tables.git"
   },
   "scripts": {
-    "update-version": "node ./scripts/update-version.js"
+    "update-version": "node ./scripts/update-version.js",
+    "pretest": "docker run --name temporal-tables-test -e POSTGRES_PASSWORD=password -e POSTGRES_USER=postgres -p 5432:5432 -d postgres:latest && sleep 3 && docker exec temporal-tables-test pg_isready -h localhost -p 5432 -U postgres -t 30",
+    "test": "(PGHOST=localhost PGPORT=5432 PGUSER=postgres PGPASSWORD=password make run_test; ret=$?; docker stop temporal-tables-test && docker rm temporal-tables-test; exit $ret)",
+    "test:nochecks": "npm run pretest && (PGHOST=localhost PGPORT=5432 PGUSER=postgres PGPASSWORD=password make run_test_nochecks; ret=$?; docker stop temporal-tables-test && docker rm temporal-tables-test; exit $ret)",
+    "cleanup": "docker stop temporal-tables-test || true && docker rm temporal-tables-test || true"
path: package.json
user: login: agubler
body: I will move it to a compose file and run that up and down, my thinking for just for a single container an inline script would suffice. As long as when running `npm test`, it ensure theres a database then that works for me. 
created_at: 2025-04-04T07:42:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,7 +7,11 @@
     "url": "git+https://github.com/nearform/temporal_tables.git"
   },
   "scripts": {
-    "update-version": "node ./scripts/update-version.js"
+    "update-version": "node ./scripts/update-version.js",
+    "pretest": "docker run --name temporal-tables-test -e POSTGRES_PASSWORD=password -e POSTGRES_USER=postgres -p 5432:5432 -d postgres:latest && sleep 3 && docker exec temporal-tables-test pg_isready -h localhost -p 5432 -U postgres -t 30",
+    "test": "(PGHOST=localhost PGPORT=5432 PGUSER=postgres PGPASSWORD=password make run_test; ret=$?; docker stop temporal-tables-test && docker rm temporal-tables-test; exit $ret)",
+    "test:nochecks": "npm run pretest && (PGHOST=localhost PGPORT=5432 PGUSER=postgres PGPASSWORD=password make run_test_nochecks; ret=$?; docker stop temporal-tables-test && docker rm temporal-tables-test; exit $ret)",
+    "cleanup": "docker stop temporal-tables-test || true && docker rm temporal-tables-test || true"
path: package.json
user: login: simoneb
body: Thanks. This script is not cross platform, wherease a docker compose file will be. The only thing we may want to mention in the dock is to spin it up for development
created_at: 2025-04-04T07:55:10Z
author_association: MEMBER
diff_hunk: @@ -7,7 +7,11 @@
     "url": "git+https://github.com/nearform/temporal_tables.git"
   },
   "scripts": {
-    "update-version": "node ./scripts/update-version.js"
+    "update-version": "node ./scripts/update-version.js",
+    "pretest": "docker run --name temporal-tables-test -e POSTGRES_PASSWORD=password -e POSTGRES_USER=postgres -p 5432:5432 -d postgres:latest && sleep 3 && docker exec temporal-tables-test pg_isready -h localhost -p 5432 -U postgres -t 30",
+    "test": "(PGHOST=localhost PGPORT=5432 PGUSER=postgres PGPASSWORD=password make run_test; ret=$?; docker stop temporal-tables-test && docker rm temporal-tables-test; exit $ret)",
+    "test:nochecks": "npm run pretest && (PGHOST=localhost PGPORT=5432 PGUSER=postgres PGPASSWORD=password make run_test_nochecks; ret=$?; docker stop temporal-tables-test && docker rm temporal-tables-test; exit $ret)",
+    "cleanup": "docker stop temporal-tables-test || true && docker rm temporal-tables-test || true"
path: package.json
user: login: agubler
body: I always forget about anything non mac 👍 
created_at: 2025-04-04T15:52:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,7 +7,11 @@
     "url": "git+https://github.com/nearform/temporal_tables.git"
   },
   "scripts": {
-    "update-version": "node ./scripts/update-version.js"
+    "update-version": "node ./scripts/update-version.js",
+    "pretest": "docker run --name temporal-tables-test -e POSTGRES_PASSWORD=password -e POSTGRES_USER=postgres -p 5432:5432 -d postgres:latest && sleep 3 && docker exec temporal-tables-test pg_isready -h localhost -p 5432 -U postgres -t 30",
+    "test": "(PGHOST=localhost PGPORT=5432 PGUSER=postgres PGPASSWORD=password make run_test; ret=$?; docker stop temporal-tables-test && docker rm temporal-tables-test; exit $ret)",
+    "test:nochecks": "npm run pretest && (PGHOST=localhost PGPORT=5432 PGUSER=postgres PGPASSWORD=password make run_test_nochecks; ret=$?; docker stop temporal-tables-test && docker rm temporal-tables-test; exit $ret)",
+    "cleanup": "docker stop temporal-tables-test || true && docker rm temporal-tables-test || true"
path: package.json
user: login: simoneb
body: Feel free to merge once you're done with this change. It's a minor thing compared to the actual feature
created_at: 2025-04-04T15:53:45Z
author_association: MEMBER

PR comments:
user: login: agubler
created_at: 2025-03-27T16:36:36Z
body: @ryan-roemer Please could you take a look at the draft blog post on copilot agents in this pull request?
author_association: CONTRIBUTOR
user: login: agubler
created_at: 2025-04-07T07:45:17Z
body: @simoneb I also need to bump the package.json version and also run the update-version script, which I've now done
author_association: CONTRIBUTOR
user: login: agubler
created_at: 2025-04-07T07:54:52Z
body: @simoneb my bad
author_association: CONTRIBUTOR
user: login: agubler
created_at: 2025-04-07T07:55:42Z
body: @simoneb done
author_association: CONTRIBUTOR
user: login: simoneb
created_at: 2025-04-07T08:01:19Z
body: Looking good, merge away!
author_association: MEMBER