Repository: celery/celery

Issue:
Database backend race condition during table creation
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
bug reports which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [ ] This has already been asked to the [discussion group](https://groups.google.com/forum/#!forum/celery-users) first.
- [x] I have read the relevant section in the
  [contribution guide](http://docs.celeryproject.org/en/latest/contributing.html#other-bugs)
  on reporting bugs.
- [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
- [x] I have checked the [commit log](https://github.com/celery/celery/commits/master)
  to find out if the bug was already fixed in the master branch.
- [x] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

## Mandatory Debugging Information

- [x] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [ ] I have included the contents of ``pip freeze`` in the issue. **N/A**
- [ ] I have included all the versions of all the external dependencies required
  to reproduce this bug. **N/A**

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->
- [ ] I have tried reproducing the issue on more than one Python version
  and/or implementation.
- [ ] I have tried reproducing the issue on more than one message broker and/or
  result backend.
- [ ] I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
- [ ] I have tried reproducing the issue on more than one operating system.
- [ ] I have tried reproducing the issue on more than one workers pool.
- [ ] I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
- [ ] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues

- https://github.com/celery/celery/issues/4653

#### Possible Duplicates

- None

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery version**: 4.4.7 (cliffs)
<!-- Include the output of celery -A proj report below -->
<details>
<summary><b><code>celery report</code> Output:</b></summary>
<p>

```
software -> celery:4.4.7 (cliffs) kombu:4.6.11 py:3.8.2
            billiard:3.6.3.0 py-amqp:2.6.1
platform -> system:Darwin arch:64bit
            kernel version:16.7.0 imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:db+postgresql://<redacted>

include: ['redacted']
accept_content: ['redacted-custom']
database_table_names: {
 'group': 'celery_group', 'task': 'celery_task'}
result_serializer: 'redacted-custom'
task_serializer: 'redacted-custom'
task_track_started: True
broker_url: 'amqp://<redacted>'
result_backend: 'db+postgresql://<redacted>'
```

</p>
</details>

# Steps to Reproduce

When celery uses a database result backend, the following line can be called multiple times from different processes:

https://github.com/celery/celery/blob/9a6c2923e859b6993227605610255bd632c1ae68/celery/backends/database/session.py#L56

This is a race condition because SQLAlchemy first checks if the tables/sequences exist and then tries to create them. It causes errors like this (at least on PostgreSQL):

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/redacted.py", line 168, in _redacted
    result = async_result.get()
  File "/usr/local/lib/python3.7/site-packages/celery/result.py", line 226, in get
    self.maybe_throw(callback=callback)
  File "/usr/local/lib/python3.7/site-packages/celery/result.py", line 342, in maybe_throw
    self.throw(value, self._to_remote_traceback(tb))
  File "/usr/local/lib/python3.7/site-packages/celery/result.py", line 335, in throw
    self.on_ready.throw(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/vine/promises.py", line 244, in throw
    reraise(type(exc), exc, tb)
  File "/usr/local/lib/python3.7/site-packages/vine/five.py", line 195, in reraise
    raise value
Exception: <class 'sqlalchemy.exc.IntegrityError'>(('(psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "pg_type_typname_nsp_index"\nDETAIL:  Key (typname, typnamespace)=(taskset_id_sequence, 2200) already exists.\n',))
```

One workaround is  to force the table creation ahead of time as was proposed by a user in the issue I linked: https://github.com/celery/celery/issues/4653#issuecomment-400029147.

I think Celery should handle this itself. A possible solution would catch `IntegrityError` and try again until `create_all` succeeds. (Perhaps with a limited number of retries and with sleeps compared to this snippet):

```python
    def prepare_models(self, engine):
        from sqlalchemy.exc import IntegrityError
        if not self.prepared:
            while True:
                try:
                    ResultModelBase.metadata.create_all(engine)
                except IntegrityError:
                    continue
                else:
                    break
            self.prepared = True
```

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.
-->

This example doesn't use celery at all, but shows that calling create_all in multiple processes can cause the error. It's a race condition, so you might need to try it multiple times or play around with the number of processes:

<details>
<p>


Requires a local postgres, and this database must be created:

```
createdb racetest
```

```python
from concurrent.futures import ProcessPoolExecutor, as_completed

from sqlalchemy import Column, Integer, Table, MetaData, create_engine

metadata = MetaData()

tbl1 = Table('tbl1', metadata, Column('id', Integer, primary_key=True))


def create_all(url):
    engine = create_engine(url)
    metadata.create_all(bind=engine)


def main():
    url = 'postgresql:///racetest'
    engine = create_engine(url)
    
    # Make sure schema is empty before we start
    metadata.drop_all(bind=engine)

    with ProcessPoolExecutor(max_workers=50) as executor:
        futures = []
        for _ in range(50):
            future = executor.submit(create_all, url)
            futures.append(future)

        for fut in as_completed(futures):
            fut.result()


if __name__ == '__main__':
    main()

```

</p>
</details>



Discussion:
you are welcome to contribute a fix for this

Resolving patch:
diff --git a/celery/backends/database/session.py b/celery/backends/database/session.py
--- a/celery/backends/database/session.py
+++ b/celery/backends/database/session.py
@@ -1,14 +1,21 @@
 """SQLAlchemy session."""
+import time
+
 from kombu.utils.compat import register_after_fork
 from sqlalchemy import create_engine
+from sqlalchemy.exc import DatabaseError
 from sqlalchemy.ext.declarative import declarative_base
 from sqlalchemy.orm import sessionmaker
 from sqlalchemy.pool import NullPool
 
+from celery.utils.time import get_exponential_backoff_interval
+
 ResultModelBase = declarative_base()
 
 __all__ = ('SessionManager',)
 
+PREPARE_MODELS_MAX_RETRIES = 10
+
 
 def _after_fork_cleanup_session(session):
     session._after_fork()
@@ -50,7 +57,25 @@ def create_session(self, dburi, short_lived_sessions=False, **kwargs):
 
     def prepare_models(self, engine):
         if not self.prepared:
-            ResultModelBase.metadata.create_all(engine)
+            # SQLAlchemy will check if the items exist before trying to
+            # create them, which is a race condition. If it raises an error
+            # in one iteration, the next may pass all the existence checks
+            # and the call will succeed.
+            retries = 0
+            while True:
+                try:
+                    ResultModelBase.metadata.create_all(engine)
+                except DatabaseError:
+                    if retries < PREPARE_MODELS_MAX_RETRIES:
+                        sleep_amount_ms = get_exponential_backoff_interval(
+                            10, retries, 1000, True
+                        )
+                        time.sleep(sleep_amount_ms / 1000)
+                        retries += 1
+                    else:
+                        raise
+                else:
+                    break
             self.prepared = True
 
     def session_factory(self, dburi, **kwargs):


Test patch:
diff --git a/t/unit/backends/test_database.py b/t/unit/backends/test_database.py
--- a/t/unit/backends/test_database.py
+++ b/t/unit/backends/test_database.py
@@ -13,7 +13,8 @@
 from celery.backends.database import (DatabaseBackend, retry, session,  # noqa
                                       session_cleanup)
 from celery.backends.database.models import Task, TaskSet  # noqa
-from celery.backends.database.session import SessionManager  # noqa
+from celery.backends.database.session import (  # noqa
+    PREPARE_MODELS_MAX_RETRIES, ResultModelBase, SessionManager)
 from t import skip  # noqa
 
 
@@ -398,3 +399,28 @@ def test_coverage_madness(self):
             SessionManager()
         finally:
             session.register_after_fork = prev
+
+    @patch('celery.backends.database.session.create_engine')
+    def test_prepare_models_terminates(self, create_engine):
+        """SessionManager.prepare_models has retry logic because the creation
+        of database tables by multiple workers is racy. This test patches
+        the used method to always raise, so we can verify that it does
+        eventually terminate.
+        """
+        from sqlalchemy.dialects.sqlite import dialect
+        from sqlalchemy.exc import DatabaseError
+
+        sqlite = dialect.dbapi()
+        manager = SessionManager()
+        engine = manager.get_engine('dburi')
+
+        def raise_err(bind):
+            raise DatabaseError("", "", [], sqlite.DatabaseError)
+
+        patch_create_all = patch.object(
+            ResultModelBase.metadata, 'create_all', side_effect=raise_err)
+
+        with pytest.raises(DatabaseError), patch_create_all as mock_create_all:
+            manager.prepare_models(engine)
+
+        assert mock_create_all.call_count == PREPARE_MODELS_MAX_RETRIES + 1