Repository: liberapay/liberapay.com

Issue:
DB deadlock during mangopay callback
https://sentry.io/share/issue/3b6782b3c362425bbd45706b70c3f024/

```
TransactionRollbackError: deadlock detected
DETAIL:  Process 22472 waits for ExclusiveLock on relation 18144 of database 16390; blocked by process 22475.
Process 22475 waits for ExclusiveLock on relation 18157 of database 16390; blocked by process 22472.
HINT:  See server log for query details.

  File "site-packages/algorithm.py", line 321, in loop
    new_state = function(**deps.as_kwargs)
  File "aspen/request_processor/algorithm.py", line 73, in render_resource
    return {'output': resource.render(state)}
  File "aspen/http/resource.py", line 88, in render
    return super(Dynamic, self).render(best_match, state)
  File "aspen/simplates/simplate.py", line 121, in render
    exec(self.pages[1], context)
  File "/opt/python/bundle/2/app/www/callbacks/mangopay.spt", line 202, in <module>
    record_exchange_result(website.db, e_id, payio.Id, status, error, p)
  File "liberapay/billing/transactions.py", line 355, in record_exchange_result
    propagate_exchange(cursor, participant, e, error, amount)
  File "liberapay/billing/transactions.py", line 439, in propagate_exchange
    merge_cash_bundles(cursor, participant.id)
  File "liberapay/billing/transactions.py", line 800, in merge_cash_bundles
    """, (p_id,))
  File "postgres/cursors.py", line 105, in one
    out = self._some(sql, parameters, lo=0, hi=1)
  File "postgres/cursors.py", line 127, in _some
    self.execute(sql, parameters)
  File "psycopg2/extras.py", line 316, in execute
    return super(NamedTupleCursor, self).execute(query, vars)
```

The deadlock timeout is 1 second (`show deadlock_timeout;` → `1s`) so I looked in the HTTP requests log a few seconds around the failed request and all I see that could have caused a deadlock is another callback:

```
[17/Nov/2017:09:16:22 +0000] 500 3190926us liberapay.com "GET /callbacks/mangopay?RessourceId=244711063&EventType=PAYIN_NORMAL_SUCCEEDED&Date=1510910149 HTTP/1.1" 243 "-"
[17/Nov/2017:09:16:23 +0000] 200 4327855us liberapay.com "GET /callbacks/mangopay?RessourceId=244342315&EventType=PAYIN_NORMAL_SUCCEEDED&Date=1510910153 HTTP/1.1" 2 "-"
```

To determine which tables were locked in a conflicting order I [looked up the relation IDs](https://stackoverflow.com/questions/3570387/postgresql-object-id-and-tuples) given in the error message:

```sql
select 18144::regclass;
   regclass   
--------------
 cash_bundles
(1 row)

select 18157::regclass;
 regclass 
----------
 takes
(1 row)
```



Discussion:
Got another two deadlocks this morning (in a group of 4 callbacks).
Several more this morning.

Resolving patch:
diff --git a/liberapay/billing/transactions.py b/liberapay/billing/transactions.py
--- a/liberapay/billing/transactions.py
+++ b/liberapay/billing/transactions.py
@@ -439,8 +439,8 @@ def propagate_exchange(cursor, participant, exchange, error, amount):
     participant.set_attributes(balance=new_balance)
 
     if amount != 0:
-        participant.update_giving_and_tippees(cursor)
         merge_cash_bundles(cursor, participant.id)
+        participant.update_giving_and_tippees(cursor)
 
 
 def transfer(db, tipper, tippee, amount, context, **kw):


Test patch: