Repository: Kinto/kinto

Issue:
PostgreSQL Cache: Deadlock
```
(psycopg2.extensions.TransactionRollbackError) deadlock detected
DETAIL:  Process 12750 waits for ShareLock on transaction 632693475; blocked by process 10080.
Process 10080 waits for ShareLock on transaction 632693479; blocked by process 12750.
HINT:  See server log for query details.
CONTEXT:  while deleting tuple (426,1) in relation "cache"
 [SQL: 'DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl;']
```


Discussion:
From what I understand about deadlocks in the official documentation, this happens when two transactions acquire locks on different objects in different order. 

Things that might be worth looking at:
* where is `cache.get()` used? Appart from pyfxa?
* In the cache backend code, the deletion of expired rows and the select are done in two separate instructions (both are executed in the same transaction, since in the same `with connect()`)

Just to check the kind of lock, in one terminal I ran:

```
[dbname] # BEGIN;
BEGIN
Time: 0,823 ms
[dbname] # DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl;
DELETE 0
Time: 23,002 ms
[dbname] # 

```

And in the other one ([source](https://wiki.postgresql.org/wiki/Lock_Monitoring))
```sql
 SELECT a.datname,
         c.relname,
         l.transactionid,
         l.mode,
         l.GRANTED,
         a.usename,
         a.query, 
         a.query_start,
         age(now(), a.query_start) AS "age", 
         a.pid 
    FROM  pg_stat_activity AS a
     JOIN pg_locks         l ON l.pid = a.pid
     JOIN pg_class         c ON c.oid = l.relation
    ORDER BY a.query_start;
```

And we can see that the `DELETE FROM cache` takes a table level lock:

```
-[ RECORD 1 ]-+---------------------------------------------------------
datname       | dbname
relname       | idx_cache_ttl
transactionid | ¤
mode          | RowExclusiveLock
granted       | t
usename       | postgres
query         | DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl;
query_start   | 2017-08-08 09:08:47.311075+00
age           | 00:02:04.741193
pid           | 29603
-[ RECORD 2 ]-+---------------------------------------------------------
datname       | dbname
relname       | cache_pkey
transactionid | ¤
mode          | RowExclusiveLock
granted       | t
usename       | postgres
query         | DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl;
query_start   | 2017-08-08 09:08:47.311075+00
age           | 00:02:04.741193
pid           | 29603
-[ RECORD 3 ]-+---------------------------------------------------------
datname       | dbname
relname       | cache
transactionid | ¤
mode          | RowExclusiveLock
granted       | t
usename       | postgres
query         | DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl;
query_start   | 2017-08-08 09:08:47.311075+00
age           | 00:02:04.741193
pid           | 29603

```


I've been able to reproduce this locally by adding some weird code to the kinto-fxa module and hammering it.

```diff
diff --git a/kinto_fxa/authentication.py b/kinto_fxa/authentication.py
index f69c421..941b204 100644
--- a/kinto_fxa/authentication.py
+++ b/kinto_fxa/authentication.py
@@ -81,6 +81,20 @@ class FxAOAuthAuthenticationPolicy(base_auth.CallbackAuthenticationPolicy):
         auth_cache = self._get_cache(request)
         auth_client = OAuthClient(server_url=server_url, cache=auth_cache)
         try:
+            #print("About to verify")
+            import random
+            import time
+            r1 = random.randint(1, 10000000)
+            cache = request.registry.cache
+            for i in range(10):
+                r1 = random.randint(1, 10000000)
+                r2 = random.randint(1, 10000000)
+                cache.set("Ethan is testing" + str(r1), "hi" + str(r2), 300)
+            try:
+                cache.get("Ethan is testing" + str(r1))
+            except Exception as e:
+                print("Exception!", r1, e)
+                raise e
             profile = auth_client.verify_token(token=auth, scope=scope)
             user_id = profile['user']
         except fxa_errors.OutOfProtocolError as e:
```

race-fxa.py:
```python
import aiohttp
import asyncio
import async_timeout

AUTH = "Bearer ......"   # some legit bearer token
URL = 'http://localhost:8888/v1/buckets/default/collections/plop'


async def fetch(session):
    with async_timeout.timeout(10000):
        async with session.get(URL) as response:
            resp = await response.json()
            print(resp)
            assert response.status == 200


async def main():
    async with aiohttp.ClientSession(headers={"Authorization": AUTH}) as session1:
        await asyncio.gather(*[fetch(session1) for i in range(4)])


loop = asyncio.get_event_loop()
loop.run_until_complete(main())
```

Run with:

```
while python race-fxa.py; date; end; date 
```

This usually produces the exception, sometimes in as few as 500 seconds, sometimes in as much as 6000 seconds. The magic numbers (ttl=300, 4 threads) seem to be relatively important because trying to change them makes it harder to reproduce (I had no luck with ttl=15 nor with ttl=3000). The exception leaves messages like this in the Postgres log:

```
ERROR:  deadlock detected
DETAIL:  Process 20392 waits for ShareLock on transaction 13130988; blocked by process 20391.
        Process 20391 waits for ShareLock on transaction 13130989; blocked by process 20392.
        Process 20392: DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl;
        Process 20391: DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl;
HINT:  See server log for query details.
CONTEXT:  while deleting tuple (482,68) in relation "cache"
STATEMENT:  DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl;
```

Finding this unhelpful, I added information to the query to make it clearer which "get" query prompted the DELETE:

```
ERROR:  deadlock detected
DETAIL:  Process 5119 waits for ShareLock on transaction 13272262; blocked by process 5117.
        Process 5117 waits for ShareLock on transaction 13272270; blocked by process 5119.
        Process 5119: DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl; -- 'fxa.oauth.verify_token:993e1f0b77d15d1ac958d4f687959eec6620cbce2684c718eaad2a4efa36513a:[''sync:addon_storage'']'
        Process 5117: DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl; -- 'Ethan is testing4822691'
HINT:  See server log for query details.
CONTEXT:  while deleting tuple (167,97) in relation "cache"
STATEMENT:  DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl; -- 'fxa.oauth.verify_token:993e1f0b77d15d1ac958d4f687959eec6620cbce2684c718eaad2a4efa36513a:[''sync:addon_storage'']'
```

As the logs indicate, both transactions are blocked on a ShareLock. Mathieu didn't find the ShareLocks for two reasons -- one, the lock doesn't show up in `pg_locks` until the second transaction is blocked on it, and two, the query he used excludes locks from `pg_locks` like this one, with `relation` being NULL. However, here's what it looks like:

```
testdb=# select * from pg_locks;
   locktype    | database | relation | page | tuple | virtualxid | transactionid | classid | objid | objsubid | virtualtransaction |  pid  |
        mode         | granted | fastpath 
---------------+----------+----------+------+-------+------------+---------------+---------+-------+----------+--------------------+-------+
---------------------+---------+----------
 relation      |    16401 |    61705 |      |       |            |               |         |       |          | 14/685             | 16268 |
 RowExclusiveLock    | t       | t
 relation      |    16401 |    61703 |      |       |            |               |         |       |          | 14/685             | 16268 |
 RowExclusiveLock    | t       | t
 relation      |    16401 |    61697 |      |       |            |               |         |       |          | 14/685             | 16268 |
 RowExclusiveLock    | t       | t
 virtualxid    |          |          |      |       | 14/685     |               |         |       |          | 14/685             | 16268 |
 ExclusiveLock       | t       | t
 relation      |    16401 |    11695 |      |       |            |               |         |       |          | 11/1007            | 21622 |
 AccessShareLock     | t       | t
 virtualxid    |          |          |      |       | 11/1007    |               |         |       |          | 11/1007            | 21622 |
 ExclusiveLock       | t       | t
 relation      |    16401 |    61705 |      |       |            |               |         |       |          | 5/2152393          | 19719 |
 RowExclusiveLock    | t       | t
 relation      |    16401 |    61703 |      |       |            |               |         |       |          | 5/2152393          | 19719 |
 RowExclusiveLock    | t       | t
 relation      |    16401 |    61697 |      |       |            |               |         |       |          | 5/2152393          | 19719 |
 RowExclusiveLock    | t       | t
 virtualxid    |          |          |      |       | 5/2152393  |               |         |       |          | 5/2152393          | 19719 |
 ExclusiveLock       | t       | t
 transactionid |          |          |      |       |            |      13272541 |         |       |          | 5/2152393          | 19719 |
 ShareLock           | f       | f       # <-- this one
 transactionid |          |          |      |       |            |      13272541 |         |       |          | 14/685             | 16268 |
 ExclusiveLock       | t       | f
 transactionid |          |          |      |       |            |      13272542 |         |       |          | 5/2152393          | 19719 |
 ExclusiveLock       | t       | f
 tuple         |    16401 |    61697 |    0 |     3 |            |               |         |       |          | 5/2152393          | 19719 |
 AccessExclusiveLock | t       | f
(14 rows)
```

From my reading (e.g. [this one](http://www.moioli.net/progetti/postgres-deadlocks-debugging-guidelines/)), the first transaction doesn't generate a lock because it can tell that the row hasn't been modified by any current transaction; instead it just optimistically updates it. The second transaction wants to grab a row lock on some rows, and this turns into a transaction lock.

It seems like a ShareLock is the kind of lock that is acquired when trying to determine whether a row matches a `WHERE` clause in some other query. So the second `DELETE` query is determining whether it's going to delete the rows deleted by the first query (which could still rollback).

A deadlock only occurs when two transactions grab locks on objects in a different order. In this case, it seems clear that the objects being locked are rows, and it seems like the queries that are grabbing locks are the `DELETE` queries. So why are the locks being grabbed in a different order? I don't 100% understand the query plans given, but it seems that the `DELETE` query is visiting rows in physical order rather than according to the index, because:

```
testdb=# explain (verbose) delete from cache where ttl is not null and '2017-08-08 19:54:06' > ttl;
                                                        QUERY PLAN                                                        
--------------------------------------------------------------------------------------------------------------------------
 Delete on public.cache  (cost=234.32..2595.19 rows=4869 width=6)
   ->  Bitmap Heap Scan on public.cache  (cost=234.32..2595.19 rows=4869 width=6)
         Output: ctid
         Recheck Cond: ((cache.ttl IS NOT NULL) AND ('2017-08-08 19:54:06'::timestamp without time zone > cache.ttl))
         ->  Bitmap Index Scan on idx_cache_ttl  (cost=0.00..233.11 rows=4869 width=0)
               Index Cond: ((cache.ttl IS NOT NULL) AND ('2017-08-08 19:54:06'::timestamp without time zone > cache.ttl))
(6 rows)
```

(A bitmap index scan is explained a bit [here](https://www.postgresql.org/docs/9.4/static/indexes-bitmap-scans.html).) Note that it generates different query plans for 'timestamp' < ttl vs. 'timestamp' > ttl, and it also generates different query plans for timestamps that are less than/greater than the entire table.

In order to explore this a little better, I've been running DELETE queries using data generated with this function:

```plpgsql
CREATE OR REPLACE FUNCTION public.insert_junk()
 RETURNS void
 LANGUAGE plpgsql
AS $function$
BEGIN
FOR i in 1 .. 100000 LOOP
  INSERT INTO cache VALUES ('Ethan is testing ' || random() * 100000, i, sec2ttl(random() * 300 + 45)) ON CONFLICT DO NOTHING;
END LOOP;
END;
$function$
```

The `value` field of the database shows its insert order, which appears to match physical order (doing a `SELECT * FROM cache LIMIT 10` shows the ones with value 1..10).

If both `DELETE` queries are locking rows in physical order, then it shouldn't be possible to induce a deadlock with just two `DELETE` queries. Therefore my suspicion, although I haven't reproduced it yet, is that one `UPDATE` is happening, then a `DELETE` is blocking on that `UPDATE`, and then a series of other `UPDATE`s are happening which change the physical order of the rows, such that when a second `DELETE` happens, it manages to grab some rows which weren't grabbed by the first `DELETE`. Then, when the first `UPDATE` commits, you get a deadlock.

I'm still trying to reproduce the thing with a clear set of queries, aided by increasingly verbose logging on my local postgres and the occasional reproduced deadlock. However, the real question is what to do about it. [Here](https://stackoverflow.com/questions/27007196/avoiding-postgresql-deadlocks-when-performing-bulk-update-and-delete-operations/27015834#27015834) is a solution that uses `SELECT FOR UPDATE` to lock rows according to an `ORDER BY` (i.e. a deterministic order that doesn't depend on physical order or change between transactions). That's probably the most targeted solution.

This is also an opportunity to rethink this part of the code more generally. @leplatrem and I both don't like that this `get()` operation, which has a very read-only sort of name, does a giant delete. To preserve correctness, we could add a `AND now() < ttl` to the `SELECT` statement to ensure we only get cache entries that haven't expired yet, and move the `DELETE` someplace else -- a "cron job" sort of thing that happens only once every N seconds, or once every N requests, or with probability xxx%, or something else entirely. (A database trigger?) Alternately, we could replace the giant `DELETE` with smaller deletes, perhaps having the `get()` method only delete outdated entries for its own key. If we can be assured that only one `DELETE` is running at a time, or that each `DELETE` only locks one row, then we don't really need to care too much about the order in which the `DELETE` locks the rows, and we wouldn't have to do the `SELECT FOR UPDATE` thing. On the other hand, it seems that we can get the error with only a few simultaneous requests, so maybe we should do the `SELECT FOR UPDATE` thing too.

It seems like writing an automated test for this would be quite difficult?

This is the most verbose log I've gotten yet (from postgres).

[broken-sql-2.txt](https://github.com/Kinto/kinto/files/1212868/broken-sql-2.txt)

This time around, the two transactions that deadlocked started (and issued their DELETE statements) before two previous DELETE statements committed. I'm not sure if that's relevant per se or if it's just part of the initial condition for the transactions/locks.

It looks like you're making progress, at least in the understanding of the lock and transactions subtleties ;)

The order of rows in the delete was a very good catch! 

Indeed, the cache index on ttl is not created on a specific order (could be `CREATE INDEX ON cache(ttl ASC)` for example) like the last_modified in the records table. And the DELETE query is not ordered either (could be `DELETE FROM cache WHERE key IN (SELECT key FROM cache ORDER BY ttl ASC` for example). 

> It seems like writing an automated test for this would be quite difficult?

Yes I agree

> To preserve correctness, we could add a AND now() < ttl to the SELECT statement to ensure we only get cache entries that haven't expired yet, and move the DELETE someplace else

Yes we could keep the `get()` as read-only, and delete old entries before adding new ones (from the `set()` method).

> This time around, the two transactions that deadlocked started (and issued their DELETE statements) before two previous DELETE statements committed. 

Since we use [SQLAlchemy sessions](http://docs.sqlalchemy.org/en/latest/core/connections.html#understanding-autocommit) in the `with client.connect()` context manager, there is always an ongoing transaction during the execution of the context manager. In the current code of `cache.get()`, a transaction covers the two operations (delete+select) which can overlap with another request.

Also, our `with_transaction=False` only means the transaction will be committed when the context manager exists, and not when the http request is served successfully.  Appart from the terrible naming of this parameter, I believe there must be a way to change that so that `with_transaction=False` would mean autocommit each `client.execute()` query. This way we would avoid this overlap of transactions for atomic operations for example.
I still can't figure out how the physical order of the rows might change, and the documentation is pretty thin on the subject. It seems like if you have a series of rows sorted by physical order, removing some of them (by updating them with a later TTL or outright deleting them) would leave you with a set that's still in physical order. https://dba.stackexchange.com/a/95852 says that DELETE can change the physical order of rows, but I don't understand how.

> Yes we could keep the get() as read-only, and delete old entries before adding new ones (from the set() method).

This wouldn't by itself address the problem; we'd also need to order the DELETE.

> Apart from the terrible naming of this parameter, I believe there must be a way to change that so that with_transaction=False would mean autocommit each client.execute() query. This way we would avoid this overlap of transactions for atomic operations for example.

I don't think the existence of a transaction is actually relevant for this bug; it's just two overlapping `DELETE` queries, both of which happen at the beginning of a transaction.

Resolving patch:
diff --git a/kinto/core/cache/postgresql/__init__.py b/kinto/core/cache/postgresql/__init__.py
--- a/kinto/core/cache/postgresql/__init__.py
+++ b/kinto/core/cache/postgresql/__init__.py
@@ -140,8 +140,17 @@ def set(self, key, value, ttl):
                                      value=value, ttl=ttl))
 
     def get(self, key):
-        purge = "DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl;"
-        query = "SELECT value FROM cache WHERE key = :key;"
+        purge = """
+        DELETE FROM cache c
+        USING (
+            SELECT key
+            FROM cache
+            WHERE ttl IS NOT NULL AND now() > ttl
+            ORDER BY key ASC
+            FOR UPDATE
+        ) del
+        WHERE del.key = c.key;"""
+        query = "SELECT value FROM cache WHERE key = :key AND now() < ttl;"
         with self.client.connect() as conn:
             conn.execute(purge)
             result = conn.execute(query, dict(key=self.prefix + key))


Test patch: