Repository: liberapay/liberapay.com

Issue:
Web server doesn't react well to DB restart
About 6 hours ago the database was restarted for the first time for an automated upgrade. Liberapay used to have problems with DB crashes when it was hosted on OpenShift so it has mechanisms to avoid failing completely and to recover automatically, but those mechanisms aren't working properly anymore. One big problem is that Liberapay is designed to run with Gunicorn, but it's currently running under Apache's mod_wsgi instead in production.


Discussion:
The server was unable to recover because Apache expects the WSGI script to be "reentrant" (as in "executable more than once in the same process"), but ours isn't:

```
[Tue Feb 19 03:52:44.735580 2019] [:error] [pid 5384] /opt/python/run/venv/local/lib/python3.6/site-packages/aspen/renderers.py:8: FutureWarning: aspen.renderers is deprecated and will be removed in a future version. Please use aspen.simplates.renderers instead.
[Tue Feb 19 03:52:44.735605 2019] [:error] [pid 5384] 'Please use aspen.simplates.renderers instead.', FutureWarning)
[Tue Feb 19 03:52:45.009172 2019] [:error] [pid 5384] INFO:botocore.credentials:Found credentials in environment variables.
[Tue Feb 19 03:52:45.193793 2019] [:error] [pid 5384] [remote 172.31.40.66:60] mod_wsgi (pid=5384): Target WSGI script '/opt/python/current/app/liberapay/main.py' cannot be loaded as Python module.
[Tue Feb 19 03:52:45.193852 2019] [:error] [pid 5384] [remote 172.31.40.66:60] mod_wsgi (pid=5384): Exception occurred processing WSGI script '/opt/python/current/app/liberapay/main.py'.
[Tue Feb 19 03:52:45.204410 2019] [:error] [pid 5384] [remote 172.31.40.66:60] Traceback (most recent call last):
[Tue Feb 19 03:52:45.204452 2019] [:error] [pid 5384] [remote 172.31.40.66:60] File "/opt/python/current/app/liberapay/main.py", line 95, in <module>
[Tue Feb 19 03:52:45.204457 2019] [:error] [pid 5384] [remote 172.31.40.66:60] d = wireup.full_algorithm.run(**website.__dict__)
[Tue Feb 19 03:52:45.204463 2019] [:error] [pid 5384] [remote 172.31.40.66:60] File "/opt/python/run/venv/local/lib/python3.6/site-packages/algorithm.py", line 342, in run
[Tue Feb 19 03:52:45.204465 2019] [:error] [pid 5384] [remote 172.31.40.66:60] loop(False)
[Tue Feb 19 03:52:45.204470 2019] [:error] [pid 5384] [remote 172.31.40.66:60] File "/opt/python/run/venv/local/lib/python3.6/site-packages/algorithm.py", line 333, in loop
[Tue Feb 19 03:52:45.204472 2019] [:error] [pid 5384] [remote 172.31.40.66:60] loop(True)
[Tue Feb 19 03:52:45.204477 2019] [:error] [pid 5384] [remote 172.31.40.66:60] File "/opt/python/run/venv/local/lib/python3.6/site-packages/algorithm.py", line 321, in loop
[Tue Feb 19 03:52:45.204479 2019] [:error] [pid 5384] [remote 172.31.40.66:60] new_state = function(**deps.as_kwargs)
[Tue Feb 19 03:52:45.204484 2019] [:error] [pid 5384] [remote 172.31.40.66:60] File "/opt/python/current/app/liberapay/wireup.py", line 156, in database
[Tue Feb 19 03:52:45.204486 2019] [:error] [pid 5384] [remote 172.31.40.66:60] oid = db.one("SELECT 'currency_amount'::regtype::oid")
[Tue Feb 19 03:52:45.204491 2019] [:error] [pid 5384] [remote 172.31.40.66:60] File "/opt/python/current/app/liberapay/wireup.py", line 113, in __getattr__
[Tue Feb 19 03:52:45.204493 2019] [:error] [pid 5384] [remote 172.31.40.66:60] raise NeedDatabase()
[Tue Feb 19 03:52:45.204508 2019] [:error] [pid 5384] [remote 172.31.40.66:60] liberapay.exceptions.NeedDatabase: 503 Service Unavailable:
[Tue Feb 19 03:52:45.208356 2019] [:error] [pid 5384] [remote 172.31.29.90:64] mod_wsgi (pid=5384): Target WSGI script '/opt/python/current/app/liberapay/main.py' cannot be loaded as Python module.
[Tue Feb 19 03:52:45.208386 2019] [:error] [pid 5384] [remote 172.31.29.90:64] mod_wsgi (pid=5384): Exception occurred processing WSGI script '/opt/python/current/app/liberapay/main.py'.
[Tue Feb 19 03:52:45.208447 2019] [:error] [pid 5384] [remote 172.31.29.90:64] Traceback (most recent call last):
[Tue Feb 19 03:52:45.208467 2019] [:error] [pid 5384] [remote 172.31.29.90:64] File "/opt/python/current/app/liberapay/main.py", line 88, in <module>
[Tue Feb 19 03:52:45.208471 2019] [:error] [pid 5384] [remote 172.31.29.90:64] del website.body_parsers[website.media_type_json]
[Tue Feb 19 03:52:45.208482 2019] [:error] [pid 5384] [remote 172.31.29.90:64] KeyError: 'application/json'
[Tue Feb 19 03:52:45.212270 2019] [:error] [pid 5384] [remote 172.31.29.90:52] mod_wsgi (pid=5384): Target WSGI script '/opt/python/current/app/liberapay/main.py' cannot be loaded as Python module.
[Tue Feb 19 03:52:45.212302 2019] [:error] [pid 5384] [remote 172.31.29.90:52] mod_wsgi (pid=5384): Exception occurred processing WSGI script '/opt/python/current/app/liberapay/main.py'.
[Tue Feb 19 03:52:45.212356 2019] [:error] [pid 5384] [remote 172.31.29.90:52] Traceback (most recent call last):
[Tue Feb 19 03:52:45.212377 2019] [:error] [pid 5384] [remote 172.31.29.90:52] File "/opt/python/current/app/liberapay/main.py", line 88, in <module>
[Tue Feb 19 03:52:45.212380 2019] [:error] [pid 5384] [remote 172.31.29.90:52] del website.body_parsers[website.media_type_json]
[Tue Feb 19 03:52:45.212392 2019] [:error] [pid 5384] [remote 172.31.29.90:52] KeyError: 'application/json'
```

This `KeyError` happened over and over again until I restarted the server with `sudo /usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf restart httpd`.

Resolving patch:
diff --git a/liberapay/main.py b/liberapay/main.py
--- a/liberapay/main.py
+++ b/liberapay/main.py
@@ -1,3 +1,13 @@
+# https://github.com/liberapay/liberapay.com/issues/1451
+import sys
+_init_modules = globals().get('_init_modules')
+if _init_modules:
+    for name in list(sys.modules):
+        if name not in _init_modules:
+            sys.modules.pop(name, None)
+else:
+    _init_modules = sys.modules.keys()
+
 import builtins
 from ipaddress import ip_address
 import os
@@ -104,9 +114,14 @@ def _assert(x):
 
 if not website.db:
     # Re-exec in 30 second to see if the DB is back up
-    # SIGTERM is used to tell gunicorn to gracefully stop the worker
-    # http://docs.gunicorn.org/en/stable/signals.html
-    Timer(30.0, lambda: os.kill(os.getpid(), signal.SIGTERM)).start()
+    if 'gunicorn' in sys.modules:
+        # SIGTERM is used to tell gunicorn to gracefully stop the worker
+        # http://docs.gunicorn.org/en/stable/signals.html
+        Timer(30.0, lambda: os.kill(os.getpid(), signal.SIGTERM)).start()
+    else:
+        # SIGUSR1 is used to tell apache to gracefully restart this worker
+        # https://httpd.apache.org/docs/current/stopping.html
+        Timer(30.0, lambda: os.kill(os.getpid(), signal.SIGUSR1)).start()
 
 if env.cache_static:
     http_caching.compile_assets(website)
diff --git a/liberapay/security/crypto.py b/liberapay/security/crypto.py
--- a/liberapay/security/crypto.py
+++ b/liberapay/security/crypto.py
@@ -89,7 +89,7 @@ def __init__(self):
             keys = secret['SecretString'].split()
         else:
             self.secrets_manager = None
-            parts = os.environ.pop('SECRET_FERNET_KEYS').split()
+            parts = os.environ['SECRET_FERNET_KEYS'].split()
             rotation_start = date(*map(int, parts[0].split('-')))
             keys = parts[1:]
         self.fernet_rotation_start = rotation_start
diff --git a/liberapay/wireup.py b/liberapay/wireup.py
--- a/liberapay/wireup.py
+++ b/liberapay/wireup.py
@@ -6,7 +6,6 @@
 import re
 import socket
 import signal
-from subprocess import call
 from time import time
 import traceback
 from urllib.request import urlretrieve
@@ -125,14 +124,9 @@ def database(env, tell_sentry):
     try:
         db = DB(dburl, maxconn=maxconn)
     except psycopg2.OperationalError as e:
-        tell_sentry(e, {})
-        pg_dir = os.environ.get('OPENSHIFT_PG_DATA_DIR')
-        if pg_dir:
-            # We know where the postgres data is, try to start the server ourselves
-            r = call(['pg_ctl', '-D', pg_dir, 'start', '-w', '-t', '15'])
-            if r == 0:
-                return database(env, tell_sentry)
+        tell_sentry(e, {}, allow_reraise=False)
         db = NoDB()
+        return {'db': db, 'db_qc1': db, 'db_qc5': db}
 
     models = (
         _AccountElsewhere, AccountElsewhere, _Community, Community,
@@ -413,6 +407,8 @@ def billing(app_conf):
 
 
 def stripe(app_conf):
+    if not app_conf:
+        return
     import stripe
     stripe.api_key = app_conf.stripe_secret_key
 
@@ -769,6 +765,8 @@ def s3(env):
 
 
 def currency_exchange_rates(db):
+    if not db:
+        return
     return {'currency_exchange_rates': get_currency_exchange_rates(db)}
 
 


Test patch:
diff --git a/liberapay/testing/emails.py b/liberapay/testing/emails.py
--- a/liberapay/testing/emails.py
+++ b/liberapay/testing/emails.py
@@ -1,4 +1,4 @@
-import mock
+from unittest import mock
 
 from liberapay.models.participant import Participant
 from liberapay.testing import Harness
diff --git a/liberapay/testing/mangopay.py b/liberapay/testing/mangopay.py
--- a/liberapay/testing/mangopay.py
+++ b/liberapay/testing/mangopay.py
@@ -1,9 +1,9 @@
 import itertools
+from unittest import mock
 
 from mangopay.resources import (
     BankAccount, CardRegistration, NaturalUser, Wallet,
 )
-import mock
 import requests
 
 from liberapay.i18n.currencies import Money
diff --git a/requirements_tests.txt b/requirements_tests.txt
--- a/requirements_tests.txt
+++ b/requirements_tests.txt
@@ -93,14 +93,6 @@ flake8==3.6.0 \
         --hash=sha256:9a7662ec724d0120012f6e29d6248ae3727d821bba522a0e6b356eff19126a49 \
         --hash=sha256:f661252913bc1dbe7fcfcbf0af0db3f42ab65aabd1a6ca68fe5d466bace94dae
 
-mock==2.0.0 \
-    --hash=sha256:5ce3c71c5545b472da17b72268978914d0252980348636840bd34a00b5cc96c1 \
-    --hash=sha256:b158b6df76edd239b8208d481dc46b6afd45a846b7812ff0ce58971cf5bc8bba
-
-    pbr==5.1.1 \
-        --hash=sha256:f6d5b23f226a2ba58e14e49aa3b1bfaf814d0199144b95d78458212444de1387 \
-        --hash=sha256:f59d71442f9ece3dffc17bc36575768e1ee9967756e6b6535f0ee1f0054c3d68
-
 vcrpy==2.0.1 \
     --hash=sha256:127e79cf7b569d071d1bd761b83f7b62b2ce2a2eb63ceca7aa67cba8f2602ea3 \
     --hash=sha256:57be64aa8e9883a4117d0b15de28af62275c001abcdb00b6dc2d4406073d9a4f
diff --git a/tests/py/test_callbacks.py b/tests/py/test_callbacks.py
--- a/tests/py/test_callbacks.py
+++ b/tests/py/test_callbacks.py
@@ -1,4 +1,4 @@
-from mock import patch
+from unittest.mock import patch
 
 from mangopay.resources import BankWirePayOut, Dispute, PayIn, Refund
 from mangopay.utils import Reason
diff --git a/tests/py/test_currencies.py b/tests/py/test_currencies.py
--- a/tests/py/test_currencies.py
+++ b/tests/py/test_currencies.py
@@ -1,6 +1,6 @@
 from decimal import Decimal as D
+from unittest.mock import patch
 
-from mock import patch
 import pytest
 
 from liberapay.billing.transactions import swap_currencies, Transfer
diff --git a/tests/py/test_elsewhere.py b/tests/py/test_elsewhere.py
--- a/tests/py/test_elsewhere.py
+++ b/tests/py/test_elsewhere.py
@@ -1,6 +1,5 @@
 import json
-
-import mock
+from unittest import mock
 
 from liberapay.billing.payday import Payday
 from liberapay.elsewhere._base import UserInfo
diff --git a/tests/py/test_email.py b/tests/py/test_email.py
--- a/tests/py/test_email.py
+++ b/tests/py/test_email.py
@@ -1,4 +1,4 @@
-from mock import patch
+from unittest.mock import patch
 
 from liberapay.exceptions import (
     BadEmailAddress, BadEmailDomain, CannotRemovePrimaryEmail, EmailAlreadyTaken,
diff --git a/tests/py/test_migrate.py b/tests/py/test_migrate.py
--- a/tests/py/test_migrate.py
+++ b/tests/py/test_migrate.py
@@ -1,7 +1,7 @@
 from os.path import abspath
+from unittest.mock import MagicMock, patch
 
 from aspen import resources
-from mock import MagicMock, patch
 
 from liberapay.constants import EPOCH
 from liberapay.testing import Harness
diff --git a/tests/py/test_pages.py b/tests/py/test_pages.py
--- a/tests/py/test_pages.py
+++ b/tests/py/test_pages.py
@@ -1,9 +1,9 @@
 from collections import OrderedDict
 import os
 import re
+from unittest.mock import patch
 
 from mangopay.resources import DirectPayIn
-from mock import patch
 from pando import json, Response
 import pytest
 
diff --git a/tests/py/test_payday.py b/tests/py/test_payday.py
--- a/tests/py/test_payday.py
+++ b/tests/py/test_payday.py
@@ -1,6 +1,5 @@
 import json
-
-import mock
+from unittest import mock
 
 from liberapay.billing.payday import create_payday_issue, main, NoPayday, Payday
 from liberapay.billing.transactions import create_debt
diff --git a/tests/py/test_payio.py b/tests/py/test_payio.py
--- a/tests/py/test_payio.py
+++ b/tests/py/test_payio.py
@@ -1,6 +1,6 @@
 from datetime import datetime
+from unittest import mock
 
-import mock
 from pando.utils import utc
 import pytest
 
diff --git a/tests/py/test_routes.py b/tests/py/test_routes.py
--- a/tests/py/test_routes.py
+++ b/tests/py/test_routes.py
@@ -1,5 +1,6 @@
+from unittest.mock import patch
+
 from mangopay.resources import Card
-from mock import patch
 
 from liberapay.testing.mangopay import MangopayHarness
 from liberapay.models.exchange_route import ExchangeRoute
diff --git a/tests/py/test_wireup.py b/tests/py/test_wireup.py
new file mode 100644
--- /dev/null
+++ b/tests/py/test_wireup.py
@@ -0,0 +1,17 @@
+import unittest
+from unittest.mock import patch
+
+
+class Test(unittest.TestCase):
+
+    def test_01_full_algorithm_doesnt_fail_when_db_is_down(self):
+        with patch.dict('os.environ', {'DATABASE_URL': 'dbname=nonexistent'}):
+            from liberapay.website import website
+            from liberapay.wireup import full_algorithm
+            full_algorithm.run(**website.__dict__)
+
+    def test_99_main_is_reentrant(self):
+        from liberapay.main import website
+        assert website
+        from liberapay.main import website
+        assert website