Repository: Kinto/kinto

Issue:
PostgreSQL timestamps bottleneck
A [long time ago](https://github.com/Kinto/kinto/commit/c6d81f7f4794a9ec2c0734a5111f84cc8d7d393a) we added the `timestamps` table, that stores the highest timestamps for each resource.

It has mainly one purpose: it allows the `ETag` obtained on an empty collection of records to be always the same until some records are created.

But the current implementation makes it a serious bottleneck. Every operation on the same resource (eg. same `(parent_id, collection_id)`) leads to an update of the same row in the `timestamps` table.

This applies to:
* creation of buckets (`(parent_id='', collection_id='bucket')`)
* creation of collections in the same bucket (`(parent_id=/buckets/{bid}, collection_id='collection')`)
* creation of records in the same collection (`(parent_id=/buckets/{bid}/collections/{cid}, collection_id='record')`)
* ...and some others like groups, accounts, ...

Since the timestamp column has a sorted index, I believe we can get rid of this denormalization and use the `timestamps` table just for the corner case of empty collections. For the nominal situation, the highest timestamp would be the max between the first row of the records index and the first row of the tombstones index.
PostgreSQL timestamps bottleneck
A [long time ago](https://github.com/Kinto/kinto/commit/c6d81f7f4794a9ec2c0734a5111f84cc8d7d393a) we added the `timestamps` table, that stores the highest timestamps for each resource.

It has mainly one purpose: it allows the `ETag` obtained on an empty collection of records to be always the same until some records are created.

But the current implementation makes it a serious bottleneck. Every operation on the same resource (eg. same `(parent_id, collection_id)`) leads to an update of the same row in the `timestamps` table.

This applies to:
* creation of buckets (`(parent_id='', collection_id='bucket')`)
* creation of collections in the same bucket (`(parent_id=/buckets/{bid}, collection_id='collection')`)
* creation of records in the same collection (`(parent_id=/buckets/{bid}/collections/{cid}, collection_id='record')`)
* ...and some others like groups, accounts, ...

Since the timestamp column has a sorted index, I believe we can get rid of this denormalization and use the `timestamps` table just for the corner case of empty collections. For the nominal situation, the highest timestamp would be the max between the first row of the records index and the first row of the tombstones index.


Discussion:

Resolving patch:
diff --git a/kinto/core/storage/memory.py b/kinto/core/storage/memory.py
--- a/kinto/core/storage/memory.py
+++ b/kinto/core/storage/memory.py
@@ -113,30 +113,34 @@ def _bump_timestamp(self, collection_id, parent_id, record=None,
         is_specified = (record is not None and
                         modified_field in record or
                         last_modified is not None)
+
+        collection_timestamp = self._timestamps[parent_id].get(collection_id, 0)
+
         if is_specified:
             # If there is a timestamp in the new record, try to use it.
             if last_modified is not None:
                 current = last_modified
             else:
                 current = record[modified_field]
-        else:
-            # Otherwise, use a new one.
-            current = utils.msec_time()
+            # If it is equal to current collection timestamp, bump it.
+            if current == collection_timestamp:
+                collection_timestamp += 1
+                current = collection_timestamp
+            # If it is superior (future), use it as new collection timestamp.
+            elif current > collection_timestamp:
+                collection_timestamp = current
+            # Else (past), do nothing.
 
-        # Bump the timestamp only if it's more than the previous one.
-        previous = self._timestamps[parent_id].get(collection_id)
-        if previous and previous >= current:
-            collection_timestamp = previous + 1
         else:
+            # Not specified, use a new one.
+            current = utils.msec_time()
+            # If two ops in the same msec, bump it.
+            if current <= collection_timestamp:
+                current = collection_timestamp + 1
             collection_timestamp = current
 
-        # In case the timestamp was specified, the collection timestamp will
-        # be different from the updated timestamp. As such, we want to return
-        # the one of the record, and not the collection one.
-        if not is_specified or previous == current:
-            current = collection_timestamp
-
         self._timestamps[parent_id][collection_id] = collection_timestamp
+
         return current
 
     @synchronized
diff --git a/kinto/core/storage/postgresql/__init__.py b/kinto/core/storage/postgresql/__init__.py
--- a/kinto/core/storage/postgresql/__init__.py
+++ b/kinto/core/storage/postgresql/__init__.py
@@ -69,7 +69,7 @@ class Storage(StorageBase):
 
     """  # NOQA
 
-    schema_version = 16
+    schema_version = 17
 
     def __init__(self, client, max_fetch_size, *args, **kwargs):
         super().__init__(*args, **kwargs)
@@ -206,28 +206,59 @@ def flush(self, auth=None):
         logger.debug('Flushed PostgreSQL storage tables')
 
     def collection_timestamp(self, collection_id, parent_id, auth=None):
-        query = """
-        WITH create_if_missing AS (
-          INSERT INTO timestamps (parent_id, collection_id, last_modified)
-          VALUES (:parent_id, :collection_id, clock_timestamp())
-          ON CONFLICT (parent_id, collection_id) DO NOTHING
-          RETURNING last_modified
-        ),
-        get_or_create AS (
-          SELECT last_modified FROM create_if_missing
+        query_existing = """
+        WITH existing_timestamps AS (
+          -- Timestamp of latest record.
+          (
+            SELECT last_modified, as_epoch(last_modified) AS last_epoch
+            FROM records
+            WHERE parent_id = :parent_id
+              AND collection_id = :collection_id
+            ORDER BY last_modified DESC
+            LIMIT 1
+          )
+          -- Timestamp of latest tombstone.
           UNION
-          SELECT last_modified FROM timestamps
-           WHERE parent_id = :parent_id
-             AND collection_id = :collection_id
+          (
+            SELECT last_modified, as_epoch(last_modified) AS last_epoch
+            FROM deleted
+            WHERE parent_id = :parent_id
+              AND collection_id = :collection_id
+            ORDER BY last_modified DESC
+            LIMIT 1
+          )
+          -- Timestamp of empty collection.
+          UNION
+          (
+            SELECT last_modified, as_epoch(last_modified) AS last_epoch
+            FROM timestamps
+            WHERE parent_id = :parent_id
+              AND collection_id = :collection_id
+          )
         )
-        SELECT as_epoch(last_modified) AS last_modified
-          FROM get_or_create;
+        SELECT MAX(last_modified) AS last_modified, MAX(last_epoch) AS last_epoch
+          FROM existing_timestamps
+        """
+
+        create_if_missing = """
+        INSERT INTO timestamps (parent_id, collection_id, last_modified)
+        VALUES (:parent_id, :collection_id, COALESCE(:last_modified, clock_timestamp()::timestamp))
+        ON CONFLICT (parent_id, collection_id) DO NOTHING
         """
+
         placeholders = dict(parent_id=parent_id, collection_id=collection_id)
         with self.client.connect(readonly=False) as conn:
-            result = conn.execute(query, placeholders)
+            existing_ts = None
+            ts_result = conn.execute(query_existing, placeholders)
+            row = ts_result.fetchone()  # Will return (None, None) when empty.
+            existing_ts = row['last_modified']
+
+            conn.execute(create_if_missing, dict(last_modified=existing_ts, **placeholders))
+
+            result = conn.execute(query_existing, placeholders)
             record = result.fetchone()
-        return record['last_modified']
+
+        return record['last_epoch']
 
     def create(self, collection_id, parent_id, record, id_generator=None,
                id_field=DEFAULT_ID_FIELD,


Test patch:
diff --git a/kinto/core/storage/testing.py b/kinto/core/storage/testing.py
--- a/kinto/core/storage/testing.py
+++ b/kinto/core/storage/testing.py
@@ -788,9 +788,10 @@ def test_create_ignores_specified_last_modified_if_in_the_past(self):
         self.assertEquals(retrieved[self.modified_field],
                           record[self.modified_field])
 
-        # Check that collection timestamp was bumped (change happened).
+        # Check that collection timestamp was not changed. Someone importing
+        # records in the past must assume the synchronization consequences.
         timestamp = self.storage.collection_timestamp(**self.storage_kw)
-        self.assertGreater(timestamp, timestamp_before)
+        self.assertEquals(timestamp, timestamp_before)
 
     def test_create_ignores_specified_last_modified_if_equal(self):
         # Create a first record, and get the timestamp.
@@ -837,9 +838,9 @@ def test_update_uses_specified_last_modified_if_in_future(self):
     def test_update_ignores_specified_last_modified_if_in_the_past(self):
         stored = self.create_record()
         record_id = stored[self.id_field]
-        timestamp_before = stored[self.modified_field]
+        timestamp_before = self.storage.collection_timestamp(**self.storage_kw)
 
-        # Set timestamp manually in the future.
+        # Set timestamp manually in the past.
         stored[self.modified_field] = timestamp_before - 10
         self.storage.update(object_id=record_id, record=stored,
                             **self.storage_kw)
@@ -850,9 +851,10 @@ def test_update_ignores_specified_last_modified_if_in_the_past(self):
         self.assertEquals(retrieved[self.modified_field],
                           stored[self.modified_field])
 
-        # Check that collection timestamp was bumped (change happened).
+        # Check that collection timestamp was not changed. Someone importing
+        # records in the past must assume the synchronization consequences.
         timestamp = self.storage.collection_timestamp(**self.storage_kw)
-        self.assertGreater(timestamp, timestamp_before)
+        self.assertEquals(timestamp, timestamp_before)
 
     def test_update_ignores_specified_last_modified_if_equal(self):
         stored = self.create_record()