Repository: celery/celery

Issue:
Django celery fixup doesn't respect Django settings for PostgreSQL connections
When using the Django-Celery fixup to run background tasks for a Django web service, the tasks in the background do not respect the settings in Django for PostgreSQL (possibly other) connections.  Every task will always create a new connection no matter the Django settings.  Although it is possible to bypass this with the environment variable CELERY_DB_REUSE_MAX, it is preferred for it to follow the settings given in Django.

## Checklist

- [ ] I have included the output of ``celery -A proj report`` in the issue.
      (if you are not able to do this, then at least specify the Celery
       version affected).
Celery 4.0.2 with potentially all versions of Django (tested on 1.10.3 and 1.11.2)
- [X] I have verified that the issue exists against the `master` branch of Celery.

This line causes this "issue":
https://github.com/celery/celery/blob/master/celery/fixups/django.py#L186

## Steps to reproduce
Note that these steps require some monitoring service to be used, we have New Relic.
Note also that we use Heroku for this app in question.
1) Have a web facing process with Django that connects to your PostgreSQL database for ORM purposes
2) Have a worker process that also connects to the PostgreSQL for ORM purposes
3) Have the DATABASES['default']['CONN_MAX_AGE'] setting set to anything that isn't 0 (easiest to see with `None` for persistent connections)
4) Make multiple requests to the web portion of Django to cause some ORM activity (easiest to see if it happens on every request)
5) Get multiple tasks to execute on the worker that will cause some ORM activity (easiest to see if it happens on every task)
6) Use your monitoring service (New Relic in our case) to view a breakdown of all of the requests and worker activity.  In New Relic you can check this using the transaction tracing; select the endpoint/task that made the db queries and check the breakdown.

## Expected behavior
psycopg2:connect would occur rarely with an average calls per transaction <<< 1

## Actual behavior
psycopg2:connect occurs very rarely with an average calls per transaction of <<< 1 for the web processes.
psycopg2:connect occurs every time with an average calls per transaction of 1 for the worker processes.

## Potential Resolution
With my limited knowledge of Celery's inner workings, it feels like a fairly simple fix that I could make on a PR myself, but I wanted some input before I spend the time setting that all up.
This fix seems to work when monkey patched into the `DjangoWorkerFixup` class.
``` Python

def _close_database(self):
    try:
        # Use Django's built in method of closing old connections.
        # This ensures that the database settings are respected.
        self._db.close_old_connections()
    except AttributeError:
        # Legacy functionality if we can't use the old connections for whatever reason.
        for conn in self._db.connections.all():
            try:
                conn.close()
            except self.interface_errors:
                pass
            except self.DatabaseError as exc:
                str_exc = str(exc)
                if 'closed' not in str_exc and 'not connected' not in str_exc:
                    raise

celery.fixups.django.DjangoWorkerFixup._close_database = _close_database
```
Django celery fixup doesn't respect Django settings for PostgreSQL connections
When using the Django-Celery fixup to run background tasks for a Django web service, the tasks in the background do not respect the settings in Django for PostgreSQL (possibly other) connections.  Every task will always create a new connection no matter the Django settings.  Although it is possible to bypass this with the environment variable CELERY_DB_REUSE_MAX, it is preferred for it to follow the settings given in Django.

## Checklist

- [ ] I have included the output of ``celery -A proj report`` in the issue.
      (if you are not able to do this, then at least specify the Celery
       version affected).
Celery 4.0.2 with potentially all versions of Django (tested on 1.10.3 and 1.11.2)
- [X] I have verified that the issue exists against the `master` branch of Celery.

This line causes this "issue":
https://github.com/celery/celery/blob/master/celery/fixups/django.py#L186

## Steps to reproduce
Note that these steps require some monitoring service to be used, we have New Relic.
Note also that we use Heroku for this app in question.
1) Have a web facing process with Django that connects to your PostgreSQL database for ORM purposes
2) Have a worker process that also connects to the PostgreSQL for ORM purposes
3) Have the DATABASES['default']['CONN_MAX_AGE'] setting set to anything that isn't 0 (easiest to see with `None` for persistent connections)
4) Make multiple requests to the web portion of Django to cause some ORM activity (easiest to see if it happens on every request)
5) Get multiple tasks to execute on the worker that will cause some ORM activity (easiest to see if it happens on every task)
6) Use your monitoring service (New Relic in our case) to view a breakdown of all of the requests and worker activity.  In New Relic you can check this using the transaction tracing; select the endpoint/task that made the db queries and check the breakdown.

## Expected behavior
psycopg2:connect would occur rarely with an average calls per transaction <<< 1

## Actual behavior
psycopg2:connect occurs very rarely with an average calls per transaction of <<< 1 for the web processes.
psycopg2:connect occurs every time with an average calls per transaction of 1 for the worker processes.

## Potential Resolution
With my limited knowledge of Celery's inner workings, it feels like a fairly simple fix that I could make on a PR myself, but I wanted some input before I spend the time setting that all up.
This fix seems to work when monkey patched into the `DjangoWorkerFixup` class.
``` Python

def _close_database(self):
    try:
        # Use Django's built in method of closing old connections.
        # This ensures that the database settings are respected.
        self._db.close_old_connections()
    except AttributeError:
        # Legacy functionality if we can't use the old connections for whatever reason.
        for conn in self._db.connections.all():
            try:
                conn.close()
            except self.interface_errors:
                pass
            except self.DatabaseError as exc:
                str_exc = str(exc)
                if 'closed' not in str_exc and 'not connected' not in str_exc:
                    raise

celery.fixups.django.DjangoWorkerFixup._close_database = _close_database
```


Discussion:
plz proceed with the PR
if it is django-celery package only then that doesn't support celery 4.x
plz proceed with the PR
if it is django-celery package only then that doesn't support celery 4.x

Resolving patch:
diff --git a/celery/fixups/django.py b/celery/fixups/django.py
--- a/celery/fixups/django.py
+++ b/celery/fixups/django.py
@@ -151,7 +151,7 @@ def on_worker_process_init(self, **kwargs):
                 self._maybe_close_db_fd(c.connection)
 
         # use the _ version to avoid DB_REUSE preventing the conn.close() call
-        self._close_database()
+        self._close_database(force=True)
         self.close_cache()
 
     def _maybe_close_db_fd(self, fd):
@@ -180,10 +180,13 @@ def close_database(self, **kwargs):
             self._close_database()
         self._db_recycles += 1
 
-    def _close_database(self):
+    def _close_database(self, force=False):
         for conn in self._db.connections.all():
             try:
-                conn.close()
+                if force:
+                    conn.close()
+                else:
+                    conn.close_if_unusable_or_obsolete()
             except self.interface_errors:
                 pass
             except self.DatabaseError as exc:


Test patch:
diff --git a/t/unit/fixups/test_django.py b/t/unit/fixups/test_django.py
--- a/t/unit/fixups/test_django.py
+++ b/t/unit/fixups/test_django.py
@@ -145,7 +145,7 @@ def test_on_worker_process_init(self, patching):
                         f.on_worker_process_init()
                         mcf.assert_called_with(conns[1].connection)
                         f.close_cache.assert_called_with()
-                        f._close_database.assert_called_with()
+                        f._close_database.assert_called_with(force=True)
 
                         f.validate_models = Mock(name='validate_models')
                         patching.setenv('FORKED_BY_MULTIPROCESSING', '1')
@@ -213,13 +213,35 @@ def test__close_database(self):
             f._db.connections = Mock()  # ConnectionHandler
             f._db.connections.all.side_effect = lambda: conns
 
-            f._close_database()
+            f._close_database(force=True)
             conns[0].close.assert_called_with()
+            conns[0].close_if_unusable_or_obsolete.assert_not_called()
             conns[1].close.assert_called_with()
+            conns[1].close_if_unusable_or_obsolete.assert_not_called()
             conns[2].close.assert_called_with()
+            conns[2].close_if_unusable_or_obsolete.assert_not_called()
+
+            for conn in conns:
+                conn.reset_mock()
+
+            f._close_database()
+            conns[0].close.assert_not_called()
+            conns[0].close_if_unusable_or_obsolete.assert_called_with()
+            conns[1].close.assert_not_called()
+            conns[1].close_if_unusable_or_obsolete.assert_called_with()
+            conns[2].close.assert_not_called()
+            conns[2].close_if_unusable_or_obsolete.assert_called_with()
 
             conns[1].close.side_effect = KeyError(
                 'omg')
+            f._close_database()
+            with pytest.raises(KeyError):
+                f._close_database(force=True)
+
+            conns[1].close.side_effect = None
+            conns[1].close_if_unusable_or_obsolete.side_effect = KeyError(
+                'omg')
+            f._close_database(force=True)
             with pytest.raises(KeyError):
                 f._close_database()