Repository: Kinto/kinto

Issue:
PostgreSQL Cache: Deadlock
```
(psycopg2.extensions.TransactionRollbackError) deadlock detected
DETAIL:  Process 12750 waits for ShareLock on transaction 632693475; blocked by process 10080.
Process 10080 waits for ShareLock on transaction 632693479; blocked by process 12750.
HINT:  See server log for query details.
CONTEXT:  while deleting tuple (426,1) in relation "cache"
 [SQL: 'DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl;']
```
Check compliance with Dockerflow spec
See https://github.com/mozilla-services/Dockerflow

* `$PORT` env variable
* Copy to `/app`
* `app` group:user
* `USER` directive
* `ENTRYPOINT` directive
* ...


Discussion:
From what I understand about deadlocks in the official documentation, this happens when two transactions acquire locks on different objects in different order. 

Things that might be worth looking at:
* where is `cache.get()` used? Appart from pyfxa?
* In the cache backend code, the deletion of expired rows and the select are done in two separate instructions (both are executed in the same transaction, since in the same `with connect()`)

Just to check the kind of lock, in one terminal I ran:

```
[dbname] # BEGIN;
BEGIN
Time: 0,823 ms
[dbname] # DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl;
DELETE 0
Time: 23,002 ms
[dbname] # 

```

And in the other one ([source](https://wiki.postgresql.org/wiki/Lock_Monitoring))
```sql
 SELECT a.datname,
         c.relname,
         l.transactionid,
         l.mode,
         l.GRANTED,
         a.usename,
         a.query, 
         a.query_start,
         age(now(), a.query_start) AS "age", 
         a.pid 
    FROM  pg_stat_activity AS a
     JOIN pg_locks         l ON l.pid = a.pid
     JOIN pg_class         c ON c.oid = l.relation
    ORDER BY a.query_start;
```

And we can see that the `DELETE FROM cache` takes a table level lock:

```
-[ RECORD 1 ]-+---------------------------------------------------------
datname       | dbname
relname       | idx_cache_ttl
transactionid | ¤
mode          | RowExclusiveLock
granted       | t
usename       | postgres
query         | DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl;
query_start   | 2017-08-08 09:08:47.311075+00
age           | 00:02:04.741193
pid           | 29603
-[ RECORD 2 ]-+---------------------------------------------------------
datname       | dbname
relname       | cache_pkey
transactionid | ¤
mode          | RowExclusiveLock
granted       | t
usename       | postgres
query         | DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl;
query_start   | 2017-08-08 09:08:47.311075+00
age           | 00:02:04.741193
pid           | 29603
-[ RECORD 3 ]-+---------------------------------------------------------
datname       | dbname
relname       | cache
transactionid | ¤
mode          | RowExclusiveLock
granted       | t
usename       | postgres
query         | DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl;
query_start   | 2017-08-08 09:08:47.311075+00
age           | 00:02:04.741193
pid           | 29603

```


I've been able to reproduce this locally by adding some weird code to the kinto-fxa module and hammering it.

```diff
diff --git a/kinto_fxa/authentication.py b/kinto_fxa/authentication.py
index f69c421..941b204 100644
--- a/kinto_fxa/authentication.py
+++ b/kinto_fxa/authentication.py
@@ -81,6 +81,20 @@ class FxAOAuthAuthenticationPolicy(base_auth.CallbackAuthenticationPolicy):
         auth_cache = self._get_cache(request)
         auth_client = OAuthClient(server_url=server_url, cache=auth_cache)
         try:
+            #print("About to verify")
+            import random
+            import time
+            r1 = random.randint(1, 10000000)
+            cache = request.registry.cache
+            for i in range(10):
+                r1 = random.randint(1, 10000000)
+                r2 = random.randint(1, 10000000)
+                cache.set("Ethan is testing" + str(r1), "hi" + str(r2), 300)
+            try:
+                cache.get("Ethan is testing" + str(r1))
+            except Exception as e:
+                print("Exception!", r1, e)
+                raise e
             profile = auth_client.verify_token(token=auth, scope=scope)
             user_id = profile['user']
         except fxa_errors.OutOfProtocolError as e:
```

race-fxa.py:
```python
import aiohttp
import asyncio
import async_timeout

AUTH = "Bearer ......"   # some legit bearer token
URL = 'http://localhost:8888/v1/buckets/default/collections/plop'


async def fetch(session):
    with async_timeout.timeout(10000):
        async with session.get(URL) as response:
            resp = await response.json()
            print(resp)
            assert response.status == 200


async def main():
    async with aiohttp.ClientSession(headers={"Authorization": AUTH}) as session1:
        await asyncio.gather(*[fetch(session1) for i in range(4)])


loop = asyncio.get_event_loop()
loop.run_until_complete(main())
```

Run with:

```
while python race-fxa.py; date; end; date 
```

This usually produces the exception, sometimes in as few as 500 seconds, sometimes in as much as 6000 seconds. The magic numbers (ttl=300, 4 threads) seem to be relatively important because trying to change them makes it harder to reproduce (I had no luck with ttl=15 nor with ttl=3000). The exception leaves messages like this in the Postgres log:

```
ERROR:  deadlock detected
DETAIL:  Process 20392 waits for ShareLock on transaction 13130988; blocked by process 20391.
        Process 20391 waits for ShareLock on transaction 13130989; blocked by process 20392.
        Process 20392: DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl;
        Process 20391: DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl;
HINT:  See server log for query details.
CONTEXT:  while deleting tuple (482,68) in relation "cache"
STATEMENT:  DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl;
```

Finding this unhelpful, I added information to the query to make it clearer which "get" query prompted the DELETE:

```
ERROR:  deadlock detected
DETAIL:  Process 5119 waits for ShareLock on transaction 13272262; blocked by process 5117.
        Process 5117 waits for ShareLock on transaction 13272270; blocked by process 5119.
        Process 5119: DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl; -- 'fxa.oauth.verify_token:993e1f0b77d15d1ac958d4f687959eec6620cbce2684c718eaad2a4efa36513a:[''sync:addon_storage'']'
        Process 5117: DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl; -- 'Ethan is testing4822691'
HINT:  See server log for query details.
CONTEXT:  while deleting tuple (167,97) in relation "cache"
STATEMENT:  DELETE FROM cache WHERE ttl IS NOT NULL AND now() > ttl; -- 'fxa.oauth.verify_token:993e1f0b77d15d1ac958d4f687959eec6620cbce2684c718eaad2a4efa36513a:[''sync:addon_storage'']'
```

As the logs indicate, both transactions are blocked on a ShareLock. Mathieu didn't find the ShareLocks for two reasons -- one, the lock doesn't show up in `pg_locks` until the second transaction is blocked on it, and two, the query he used excludes locks from `pg_locks` like this one, with `relation` being NULL. However, here's what it looks like:

```
testdb=# select * from pg_locks;
   locktype    | database | relation | page | tuple | virtualxid | transactionid | classid | objid | objsubid | virtualtransaction |  pid  |
        mode         | granted | fastpath 
---------------+----------+----------+------+-------+------------+---------------+---------+-------+----------+--------------------+-------+
---------------------+---------+----------
 relation      |    16401 |    61705 |      |       |            |               |         |       |          | 14/685             | 16268 |
 RowExclusiveLock    | t       | t
 relation      |    16401 |    61703 |      |       |            |               |         |       |          | 14/685             | 16268 |
 RowExclusiveLock    | t       | t
 relation      |    16401 |    61697 |      |       |            |               |         |       |          | 14/685             | 16268 |
 RowExclusiveLock    | t       | t
 virtualxid    |          |          |      |       | 14/685     |               |         |       |          | 14/685             | 16268 |
 ExclusiveLock       | t       | t
 relation      |    16401 |    11695 |      |       |            |               |         |       |          | 11/1007            | 21622 |
 AccessShareLock     | t       | t
 virtualxid    |          |          |      |       | 11/1007    |               |         |       |          | 11/1007            | 21622 |
 ExclusiveLock       | t       | t
 relation      |    16401 |    61705 |      |       |            |               |         |       |          | 5/2152393          | 19719 |
 RowExclusiveLock    | t       | t
 relation      |    16401 |    61703 |      |       |            |               |         |       |          | 5/2152393          | 19719 |
 RowExclusiveLock    | t       | t
 relation      |    16401 |    61697 |      |       |            |               |         |       |          | 5/2152393          | 19719 |
 RowExclusiveLock    | t       | t
 virtualxid    |          |          |      |       | 5/2152393  |               |         |       |          | 5/2152393          | 19719 |
 ExclusiveLock       | t       | t
 transactionid |          |          |      |       |            |      13272541 |         |       |          | 5/2152393          | 19719 |
 ShareLock           | f       | f       # <-- this one
 transactionid |          |          |      |       |            |      13272541 |         |       |          | 14/685             | 16268 |
 ExclusiveLock       | t       | f
 transactionid |          |          |      |       |            |      13272542 |         |       |          | 5/2152393          | 19719 |
 ExclusiveLock       | t       | f
 tuple         |    16401 |    61697 |    0 |     3 |            |               |         |       |          | 5/2152393          | 19719 |
 AccessExclusiveLock | t       | f
(14 rows)
```

From my reading (e.g. [this one](http://www.moioli.net/progetti/postgres-deadlocks-debugging-guidelines/)), the first transaction doesn't generate a lock because it can tell that the row hasn't been modified by any current transaction; instead it just optimistically updates it. The second transaction wants to grab a row lock on some rows, and this turns into a transaction lock.

It seems like a ShareLock is the kind of lock that is acquired when trying to determine whether a row matches a `WHERE` clause in some other query. So the second `DELETE` query is determining whether it's going to delete the rows deleted by the first query (which could still rollback).

A deadlock only occurs when two transactions grab locks on objects in a different order. In this case, it seems clear that the objects being locked are rows, and it seems like the queries that are grabbing locks are the `DELETE` queries. So why are the locks being grabbed in a different order? I don't 100% understand the query plans given, but it seems that the `DELETE` query is visiting rows in physical order rather than according to the index, because:

```
testdb=# explain (verbose) delete from cache where ttl is not null and '2017-08-08 19:54:06' > ttl;
                                                        QUERY PLAN                                                        
--------------------------------------------------------------------------------------------------------------------------
 Delete on public.cache  (cost=234.32..2595.19 rows=4869 width=6)
   ->  Bitmap Heap Scan on public.cache  (cost=234.32..2595.19 rows=4869 width=6)
         Output: ctid
         Recheck Cond: ((cache.ttl IS NOT NULL) AND ('2017-08-08 19:54:06'::timestamp without time zone > cache.ttl))
         ->  Bitmap Index Scan on idx_cache_ttl  (cost=0.00..233.11 rows=4869 width=0)
               Index Cond: ((cache.ttl IS NOT NULL) AND ('2017-08-08 19:54:06'::timestamp without time zone > cache.ttl))
(6 rows)
```

(A bitmap index scan is explained a bit [here](https://www.postgresql.org/docs/9.4/static/indexes-bitmap-scans.html).) Note that it generates different query plans for 'timestamp' < ttl vs. 'timestamp' > ttl, and it also generates different query plans for timestamps that are less than/greater than the entire table.

In order to explore this a little better, I've been running DELETE queries using data generated with this function:

```plpgsql
CREATE OR REPLACE FUNCTION public.insert_junk()
 RETURNS void
 LANGUAGE plpgsql
AS $function$
BEGIN
FOR i in 1 .. 100000 LOOP
  INSERT INTO cache VALUES ('Ethan is testing ' || random() * 100000, i, sec2ttl(random() * 300 + 45)) ON CONFLICT DO NOTHING;
END LOOP;
END;
$function$
```

The `value` field of the database shows its insert order, which appears to match physical order (doing a `SELECT * FROM cache LIMIT 10` shows the ones with value 1..10).

If both `DELETE` queries are locking rows in physical order, then it shouldn't be possible to induce a deadlock with just two `DELETE` queries. Therefore my suspicion, although I haven't reproduced it yet, is that one `UPDATE` is happening, then a `DELETE` is blocking on that `UPDATE`, and then a series of other `UPDATE`s are happening which change the physical order of the rows, such that when a second `DELETE` happens, it manages to grab some rows which weren't grabbed by the first `DELETE`. Then, when the first `UPDATE` commits, you get a deadlock.

I'm still trying to reproduce the thing with a clear set of queries, aided by increasingly verbose logging on my local postgres and the occasional reproduced deadlock. However, the real question is what to do about it. [Here](https://stackoverflow.com/questions/27007196/avoiding-postgresql-deadlocks-when-performing-bulk-update-and-delete-operations/27015834#27015834) is a solution that uses `SELECT FOR UPDATE` to lock rows according to an `ORDER BY` (i.e. a deterministic order that doesn't depend on physical order or change between transactions). That's probably the most targeted solution.

This is also an opportunity to rethink this part of the code more generally. @leplatrem and I both don't like that this `get()` operation, which has a very read-only sort of name, does a giant delete. To preserve correctness, we could add a `AND now() < ttl` to the `SELECT` statement to ensure we only get cache entries that haven't expired yet, and move the `DELETE` someplace else -- a "cron job" sort of thing that happens only once every N seconds, or once every N requests, or with probability xxx%, or something else entirely. (A database trigger?) Alternately, we could replace the giant `DELETE` with smaller deletes, perhaps having the `get()` method only delete outdated entries for its own key. If we can be assured that only one `DELETE` is running at a time, or that each `DELETE` only locks one row, then we don't really need to care too much about the order in which the `DELETE` locks the rows, and we wouldn't have to do the `SELECT FOR UPDATE` thing. On the other hand, it seems that we can get the error with only a few simultaneous requests, so maybe we should do the `SELECT FOR UPDATE` thing too.

It seems like writing an automated test for this would be quite difficult?

This is the most verbose log I've gotten yet (from postgres).

[broken-sql-2.txt](https://github.com/Kinto/kinto/files/1212868/broken-sql-2.txt)

This time around, the two transactions that deadlocked started (and issued their DELETE statements) before two previous DELETE statements committed. I'm not sure if that's relevant per se or if it's just part of the initial condition for the transactions/locks.

It looks like you're making progress, at least in the understanding of the lock and transactions subtleties ;)

The order of rows in the delete was a very good catch! 

Indeed, the cache index on ttl is not created on a specific order (could be `CREATE INDEX ON cache(ttl ASC)` for example) like the last_modified in the records table. And the DELETE query is not ordered either (could be `DELETE FROM cache WHERE key IN (SELECT key FROM cache ORDER BY ttl ASC` for example). 

> It seems like writing an automated test for this would be quite difficult?

Yes I agree

> To preserve correctness, we could add a AND now() < ttl to the SELECT statement to ensure we only get cache entries that haven't expired yet, and move the DELETE someplace else

Yes we could keep the `get()` as read-only, and delete old entries before adding new ones (from the `set()` method).

> This time around, the two transactions that deadlocked started (and issued their DELETE statements) before two previous DELETE statements committed. 

Since we use [SQLAlchemy sessions](http://docs.sqlalchemy.org/en/latest/core/connections.html#understanding-autocommit) in the `with client.connect()` context manager, there is always an ongoing transaction during the execution of the context manager. In the current code of `cache.get()`, a transaction covers the two operations (delete+select) which can overlap with another request.

Also, our `with_transaction=False` only means the transaction will be committed when the context manager exists, and not when the http request is served successfully.  Appart from the terrible naming of this parameter, I believe there must be a way to change that so that `with_transaction=False` would mean autocommit each `client.execute()` query. This way we would avoid this overlap of transactions for atomic operations for example.
I still can't figure out how the physical order of the rows might change, and the documentation is pretty thin on the subject. It seems like if you have a series of rows sorted by physical order, removing some of them (by updating them with a later TTL or outright deleting them) would leave you with a set that's still in physical order. https://dba.stackexchange.com/a/95852 says that DELETE can change the physical order of rows, but I don't understand how.

> Yes we could keep the get() as read-only, and delete old entries before adding new ones (from the set() method).

This wouldn't by itself address the problem; we'd also need to order the DELETE.

> Apart from the terrible naming of this parameter, I believe there must be a way to change that so that with_transaction=False would mean autocommit each client.execute() query. This way we would avoid this overlap of transactions for atomic operations for example.

I don't think the existence of a transaction is actually relevant for this bug; it's just two overlapping `DELETE` queries, both of which happen at the beginning of a transaction.
Based on some comments on my [StackOverflow cry for help](https://stackoverflow.com/questions/45621347/how-can-two-delete-queries-deadlock-in-postgres), it seems like actually, the index *is* being used, and it's guiding the lock acquisition. In #1310, I tried locking them in `ttl` order, and that lead to deadlocks. Locking them according to key (`key`), as suggested by a commenter, leads to no deadlocks. So does locking them according to physical order (`ctid`).

I still don't really understand why locking them according to `ttl` order leads to deadlocks in our application. Any update to a cache entry should remove it from the set that should be about to be deleted. One commenter suggested that ordering by `ttl` is not well-defined if two entries have the same `ttl`, but I haven't been able to reproduce it that way either. But at this point I'm running out of patience -- I have two fixes that both seem to work and a vague understanding, so maybe that's good enough.
OK, I did a little bit more investigation this morning. I set the `deadlock_timeout` parameter to one day, and then when the transactions deadlocked I was able to use SQL queries to investigate what rows had been grabbed by what transactions (using `SELECT xmax, * FROM cache ORDER BY ttl limit 10`). It seems like the two transactions deadlocked by both grabbing one of two rows that have the same `ttl`. I guess having equal `ttl` means that the order is undefined. Even with this information, I still wasn't able to reproduce the deadlock "by hand" -- in my tests, it always appears to grab the locks in a deterministic way (I guess by `ttl` and then physical order, or something like that). Still, #1312 should address the problem.


Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -71,9 +71,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '7.0'
+version = '7.3'
 # The full version, including alpha/beta/rc tags.
-release = '7.3.1'
+release = '7.3.2'
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -77,7 +77,7 @@ def read_file(filename):
 
 
 setup(name='kinto',
-      version='7.3.2.dev0',
+      version='7.3.2',
       description='Kinto Web Service - Store, Sync, Share, and Self-Host.',
       long_description="{}\n\n{}\n\n{}".format(README, CHANGELOG, CONTRIBUTORS),
       license='Apache License (2.0)',


Test patch: