Issue 1:
When the first task sent by a client on a fresh celery setup is a group task (or possibly any set of tasks processed concurrently by multiple workers), and PostgreSQL is used as the result backend, the workers fail with an IntegrityError due to a duplicate key.

Steps to reproduce

Send a group task to multiple workers as the first task in a new setup

Retrieve the group result

Expected behavior

The workers are able to store their result at the PostgreSQL backend, and the group result is correctly retrieved by the client.

Actual behavior

The _store_result operation fails on the workers with an IntegrityError. The operation is retried three times, but sometimes this is not sufficient and the group task fails with the IntegrityError.

Based on the traceback, my suspicion is that as multiple workers are concurrently processing a task and ultimately want to store their result at the backend, they are concurrently setting up sessions with the database:

celery/celery/backends/database/__init__.py

Lines 95 to 105
in
02c977d

def ResultSession(self, session_manager=SessionManager()):

return session_manager.session_factory(

dburi=self.url,

short_lived_sessions=self.short_lived_sessions,

**self.engine_options)

@retry

def _store_result(self, task_id, result, state,

traceback=None, max_retries=3, **kwargs):

"""Store return value and state of an executed task."""

session = self.ResultSession()

As it is the first task the workers process, this entails preparing the models and creating the necessary tables with create_all (using SQLAlchemy):

celery/celery/backends/database/session.py

Lines 53 to 60
in
02c977d

def prepare_models(self, engine):

if not self.prepared:

ResultModelBase.metadata.create_all(engine)

self.prepared = True

def session_factory(self, dburi, **kwargs):

engine, session = self.create_session(dburi, **kwargs)

self.prepare_models(engine)

It seems PostgreSQL doesn't handle multiple sessions creating the same table very well (1,2,3), so a duplicate key in an index for those tables cause an IntegrityError.

#4267 could be an issue with a similar cause.

Traceback

[2018-04-10 13:46:40,433: WARNING/ForkPoolWorker-1] Failed operation _store_result. Retrying 2 more times.
Traceback (most recent call last):
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
cursor.execute(statement, parameters)
psycopg2.IntegrityError: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
DETAIL: Key (typname, typnamespace)=(celery_taskmeta, 2200) already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 53, in _inner
return fun(*args, **kwargs)
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 105, in _store_result
session = self.ResultSession()
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 99, in ResultSession
**self.engine_options)
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/session.py", line 59, in session_factory
self.prepare_models(engine)
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/session.py", line 54, in prepare_models
ResultModelBase.metadata.create_all(engine)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/schema.py", line 4004, in create_all
tables=tables)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1940, in _run_visitor
conn._run_visitor(visitorcallable, element, **kwargs)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1549, in _run_visitor
**kwargs).traverse_single(element)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
return meth(obj, **kw)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 757, in visit_metadata
_is_metadata_operation=True)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
return meth(obj, **kw)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 791, in visit_table
include_foreign_key_constraints=include_foreign_key_constraints
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 948, in execute
return meth(self, multiparams, params)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
return connection._execute_ddl(self, multiparams, params)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1009, in _execute_ddl
compiled
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
context)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
exc_info
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
raise value.with_traceback(tb)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) duplicate key value violates unique constraint "pg_type_typname_nsp_index"
DETAIL: Key (typname, typnamespace)=(celery_taskmeta, 2200) already exists.
[SQL: '\nCREATE TABLE celery_taskmeta (\n\tid INTEGER NOT NULL, \n\ttask_id VARCHAR(155), \n\tstatus VARCHAR(50), \n\tresult BYTEA, \n\tdate_done TIMESTAMP WITHOUT TIME ZONE, \n\ttraceback TEXT, \n\tPRIMARY KEY (id), \n\tUNIQUE (task_id)\n)\n\n'] (Background on this error at: http://sqlalche.me/e/gkpj)
[2018-04-10 13:46:40,575: WARNING/ForkPoolWorker-1] Failed operation _store_result. Retrying 1 more times.
Traceback (most recent call last):
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
cursor.execute(statement, parameters)
psycopg2.IntegrityError: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
DETAIL: Key (typname, typnamespace)=(taskset_id_sequence, 2200) already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 53, in _inner
return fun(*args, **kwargs)
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 105, in _store_result
session = self.ResultSession()
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 99, in ResultSession
**self.engine_options)
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/session.py", line 59, in session_factory
self.prepare_models(engine)
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/session.py", line 54, in prepare_models
ResultModelBase.metadata.create_all(engine)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/schema.py", line 4004, in create_all
tables=tables)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1940, in _run_visitor
conn._run_visitor(visitorcallable, element, **kwargs)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1549, in _run_visitor
**kwargs).traverse_single(element)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
return meth(obj, **kw)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 757, in visit_metadata
_is_metadata_operation=True)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
return meth(obj, **kw)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 782, in visit_table
self.traverse_single(column.default)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
return meth(obj, **kw)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 820, in visit_sequence
self.connection.execute(CreateSequence(sequence))
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 948, in execute
return meth(self, multiparams, params)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
return connection._execute_ddl(self, multiparams, params)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1009, in _execute_ddl
compiled
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
context)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
exc_info
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
raise value.with_traceback(tb)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) duplicate key value violates unique constraint "pg_type_typname_nsp_index"
DETAIL: Key (typname, typnamespace)=(taskset_id_sequence, 2200) already exists.
[SQL: 'CREATE SEQUENCE taskset_id_sequence'] (Background on this error at: http://sqlalche.me/e/gkpj)
[2018-04-10 13:46:40,670: WARNING/ForkPoolWorker-1] Failed operation _store_result. Retrying 0 more times.
Traceback (most recent call last):
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
cursor.execute(statement, parameters)
psycopg2.IntegrityError: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
DETAIL: Key (typname, typnamespace)=(celery_tasksetmeta, 2200) already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 53, in _inner
return fun(*args, **kwargs)
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 105, in _store_result
session = self.ResultSession()
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 99, in ResultSession
**self.engine_options)
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/session.py", line 59, in session_factory
self.prepare_models(engine)
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/session.py", line 54, in prepare_models
ResultModelBase.metadata.create_all(engine)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/schema.py", line 4004, in create_all
tables=tables)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1940, in _run_visitor
conn._run_visitor(visitorcallable, element, **kwargs)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1549, in _run_visitor
**kwargs).traverse_single(element)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
return meth(obj, **kw)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 757, in visit_metadata
_is_metadata_operation=True)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
return meth(obj, **kw)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 791, in visit_table
include_foreign_key_constraints=include_foreign_key_constraints
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 948, in execute
return meth(self, multiparams, params)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
return connection._execute_ddl(self, multiparams, params)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1009, in _execute_ddl
compiled
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
context)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
exc_info
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
raise value.with_traceback(tb)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) duplicate key value violates unique constraint "pg_type_typname_nsp_index"
DETAIL: Key (typname, typnamespace)=(celery_tasksetmeta, 2200) already exists.
[SQL: '\nCREATE TABLE celery_tasksetmeta (\n\tid INTEGER NOT NULL, \n\ttaskset_id VARCHAR(155), \n\tresult BYTEA, \n\tdate_done TIMESTAMP WITHOUT TIME ZONE, \n\tPRIMARY KEY (id), \n\tUNIQUE (taskset_id)\n)\n\n'] (Background on this error at: http://sqlalche.me/e/gkpj)
[2018-04-10 13:46:40,675: WARNING/ForkPoolWorker-1] /home/ubuntu/venv/lib/python3.6/site-packages/celery/app/trace.py:549: RuntimeWarning: Exception raised outside body: IntegrityError('(psycopg2.IntegrityError) duplicate key value violates unique constraint "pg_type_typname_nsp_index"\nDETAIL: Key (typname, typnamespace)=(celery_tasksetmeta, 2200) already exists.\n',):
Traceback (most recent call last):
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
cursor.execute(statement, parameters)
psycopg2.IntegrityError: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
DETAIL: Key (typname, typnamespace)=(celery_tasksetmeta, 2200) already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/app/trace.py", line 434, in trace_task
uuid, retval, task_request, publish_result,
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/base.py", line 152, in mark_as_done
self.store_result(task_id, result, state, request=request)
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/base.py", line 309, in store_result
request=request, **kwargs)
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 53, in _inner
return fun(*args, **kwargs)
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 105, in _store_result
session = self.ResultSession()
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 99, in ResultSession
**self.engine_options)
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/session.py", line 59, in session_factory
self.prepare_models(engine)
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/session.py", line 54, in prepare_models
ResultModelBase.metadata.create_all(engine)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/schema.py", line 4004, in create_all
tables=tables)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1940, in _run_visitor
conn._run_visitor(visitorcallable, element, **kwargs)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1549, in _run_visitor
**kwargs).traverse_single(element)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
return meth(obj, **kw)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 757, in visit_metadata
_is_metadata_operation=True)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
return meth(obj, **kw)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 791, in visit_table
include_foreign_key_constraints=include_foreign_key_constraints
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 948, in execute
return meth(self, multiparams, params)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
return connection._execute_ddl(self, multiparams, params)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1009, in _execute_ddl
compiled
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
context)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
exc_info
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
raise value.with_traceback(tb)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) duplicate key value violates unique constraint "pg_type_typname_nsp_index"
DETAIL: Key (typname, typnamespace)=(celery_tasksetmeta, 2200) already exists.
[SQL: '\nCREATE TABLE celery_tasksetmeta (\n\tid INTEGER NOT NULL, \n\ttaskset_id VARCHAR(155), \n\tresult BYTEA, \n\tdate_done TIMESTAMP WITHOUT TIME ZONE, \n\tPRIMARY KEY (id), \n\tUNIQUE (taskset_id)\n)\n\n'] (Background on this error at: http://sqlalche.me/e/gkpj)

exc, exc_info.traceback)))
[2018-04-10 13:46:40,712: ERROR/ForkPoolWorker-1] Task setup[cfe2557a-f9a1-4406-86d0-1499166cc1a6] raised unexpected: IntegrityError('(psycopg2.IntegrityError) duplicate key value violates unique constraint "pg_type_typname_nsp_index"\nDETAIL: Key (typname, typnamespace)=(celery_tasksetmeta, 2200) already exists.\n',)
Traceback (most recent call last):
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
cursor.execute(statement, parameters)
psycopg2.IntegrityError: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
DETAIL: Key (typname, typnamespace)=(celery_tasksetmeta, 2200) already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/app/trace.py", line 434, in trace_task
uuid, retval, task_request, publish_result,
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/base.py", line 152, in mark_as_done
self.store_result(task_id, result, state, request=request)
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/base.py", line 309, in store_result
request=request, **kwargs)
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 53, in _inner
return fun(*args, **kwargs)
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 105, in _store_result
session = self.ResultSession()
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/__init__.py", line 99, in ResultSession
**self.engine_options)
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/session.py", line 59, in session_factory
self.prepare_models(engine)
File "/home/ubuntu/venv/lib/python3.6/site-packages/celery/backends/database/session.py", line 54, in prepare_models
ResultModelBase.metadata.create_all(engine)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/schema.py", line 4004, in create_all
tables=tables)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1940, in _run_visitor
conn._run_visitor(visitorcallable, element, **kwargs)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1549, in _run_visitor
**kwargs).traverse_single(element)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
return meth(obj, **kw)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 757, in visit_metadata
_is_metadata_operation=True)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
return meth(obj, **kw)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 791, in visit_table
include_foreign_key_constraints=include_foreign_key_constraints
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 948, in execute
return meth(self, multiparams, params)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
return connection._execute_ddl(self, multiparams, params)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1009, in _execute_ddl
compiled
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
context)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
exc_info
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
raise value.with_traceback(tb)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/home/ubuntu/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) duplicate key value violates unique constraint "pg_type_typname_nsp_index"
DETAIL: Key (typname, typnamespace)=(celery_tasksetmeta, 2200) already exists.
[SQL: '\nCREATE TABLE celery_tasksetmeta (\n\tid INTEGER NOT NULL, \n\ttaskset_id VARCHAR(155), \n\tresult BYTEA, \n\tdate_done TIMESTAMP WITHOUT TIME ZONE, \n\tPRIMARY KEY (id), \n\tUNIQUE (taskset_id)\n)\n\n'] (Background on this error at: http://sqlalche.me/e/gkpj)

Report

software -> celery:4.2.0rc2 (windowlicker) kombu:4.1.0 py:3.6.3
billiard:3.5.0.3 py-amqp:2.2.2
platform -> system:Linux arch:64bit, ELF imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:db+postgresql://****

SQLAlchemy version 1.2.6

Issue 2:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected). 5.0.0

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version: 5.0.0

celery report Output:

Steps to Reproduce

Try to install celery 5.0.0 now that click released version 8.0 - it will install click 8.0 and be broken since click_repl 0.1.6 does not work with click 8.0

When celery app is then started it produces this output:

/home/tomasz_kluczkowski/.virtualenvs/celery-experiments/bin/python /home/tomasz_kluczkowski/.virtualenvs/celery-experiments/bin/celery -A tasks worker -l INFO -E -c 4
Traceback (most recent call last):
File "/home/tomasz_kluczkowski/.virtualenvs/celery-experiments/bin/celery", line 8, in <module>
sys.exit(main())
File "/home/tomasz_kluczkowski/.virtualenvs/celery-experiments/lib/python3.9/site-packages/celery/__main__.py", line 14, in main
from celery.bin.celery import main as _main
File "/home/tomasz_kluczkowski/.virtualenvs/celery-experiments/lib/python3.9/site-packages/celery/bin/celery.py", line 10, in <module>
from celery.bin.amqp import amqp
File "/home/tomasz_kluczkowski/.virtualenvs/celery-experiments/lib/python3.9/site-packages/celery/bin/amqp.py", line 7, in <module>
from click_repl import register_repl
File "/home/tomasz_kluczkowski/.virtualenvs/celery-experiments/lib/python3.9/site-packages/click_repl/__init__.py", line 6, in <module>
import click._bashcomplete
ModuleNotFoundError: No module named 'click._bashcomplete'

Process finished with exit code 1

Required Dependencies

Minimal Python Version: I tried on 3.9, but it has no influence

Minimal Celery Version: 5.0.0 exact

Minimal Kombu Version: N/A

Minimal Broker Version: N/A

Minimal Result Backend Version: N/A

Minimal OS and/or Kernel Version: N/A

Minimal Broker Client Version: N/A

Minimal Result Backend Client Version: N/A

Python Packages

pip freeze Output:

amqp==5.0.5
billiard==3.6.3.0
celery==5.0.0
click==8.0.0
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.1.6
flower==1.0.1
humanize==0.5.1
kombu==5.0.2
prometheus-client==0.10.1
prompt-toolkit==3.0.17
pytz==2021.1
redis==3.5.3
six==1.15.0
tornado==6.1
vine==5.0.0
wcwidth==0.2.5

Other Dependencies

N/A

Minimally Reproducible Test Case

pip uninstall celery click
pip install celery==5

this will now pull celery 5 plus click 8.0 (and all the other stuff celery needs, only click 8.0 is to be fixed)

Expected Behavior

celery requires click version which can actually be used with click_repl

Actual Behavior

exception on starting celery app

Starting celery 5.0.0 with click 8.0 produces an exception as below:

/home/tomasz_kluczkowski/.virtualenvs/celery-experiments/bin/python /home/tomasz_kluczkowski/.virtualenvs/celery-experiments/bin/celery -A tasks worker -l INFO -E -c 4
Traceback (most recent call last):
File "/home/tomasz_kluczkowski/.virtualenvs/celery-experiments/bin/celery", line 8, in <module>
sys.exit(main())
File "/home/tomasz_kluczkowski/.virtualenvs/celery-experiments/lib/python3.9/site-packages/celery/__main__.py", line 14, in main
from celery.bin.celery import main as _main
File "/home/tomasz_kluczkowski/.virtualenvs/celery-experiments/lib/python3.9/site-packages/celery/bin/celery.py", line 10, in <module>
from celery.bin.amqp import amqp
File "/home/tomasz_kluczkowski/.virtualenvs/celery-experiments/lib/python3.9/site-packages/celery/bin/amqp.py", line 7, in <module>
from click_repl import register_repl
File "/home/tomasz_kluczkowski/.virtualenvs/celery-experiments/lib/python3.9/site-packages/click_repl/__init__.py", line 6, in <module>
import click._bashcomplete
ModuleNotFoundError: No module named 'click._bashcomplete'

Process finished with exit code 1

This can be easily fixed by changing requirements for 5.0.0 tag for click to be same as on 5.0.5: click >=7, <8