Issue 1:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

[ x I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

[-] I have tried reproducing the issue on more than one Python version

and/or implementation.

[-] I have tried reproducing the issue on more than one message broker and/or

result backend.

[-] I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

[-] I have tried reproducing the issue on more than one operating system.

[-] I have tried reproducing the issue on more than one workers pool.

[-] I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

[-] I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

4.3.0 (rhubarb)

Celery version:

celery report Output:

software -> celery:4.3.0 (rhubarb) kombu:4.6.1 py:3.6.9
billiard:3.6.0.0 redis:3.2.1
platform -> system:Darwin arch:64bit
kernel version:17.7.0 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://localhost:6379/0

accept_content: ['application/json',
'application/x-python-serialize',
'application/x-bson']
broker_heartbeat: 0
broker_url: 'redis://localhost:6379/0'
enable_utc: True
imports:
('service.tasks',)

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.6.9

Minimal Celery Version: 4.3

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

alembic==1.0.10
amqp==2.5.0
aniso8601==6.0.0
billiard==3.6.0.0
boto3==1.9.162
botocore==1.12.162
bson==0.5.8
celery==4.3.0
certifi==2019.3.9
chardet==3.0.4
Click==7.0
docutils==0.14
Flask==1.0.3
Flask-RESTful==0.3.7
gevent==1.4.0
greenlet==0.4.15
grequests==0.3.0
idna==2.7
itsdangerous==1.1.0
Jinja2==2.10.1
jmespath==0.9.4
kombu==4.6.1
Mako==1.0.12
MarkupSafe==1.1.1
pg8000==1.12.3
psycopg2==2.7.5
pylru==1.2.0
python-dateutil==2.8.0
python-editor==1.0.4
python-json-logger==0.1.11
pytz==2019.1
redis==3.2.1
requests==2.19.1
s3transfer==0.2.1
six==1.12.0
SQLAlchemy==1.3.4
urllib3==1.23
vine==1.3.0
Werkzeug==0.15.4

Other Dependencies

N/A

Minimally Reproducible Test Case

a chord over redis broker, one of the tasks raises exception and the error is not registered correctly in celery. the exception is as follows:

[r-celery-worker-cpu-5d47cffb98-n86dz] [2019-11-21 14:47:26,053: ERROR/ForkPoolWorker-1] Chord callback for 'b30c24c5-9653-49bf-8917-87742b82155c' raised: TypeError('__init__() takes 1 positional argument but 2 were given',)
[r-celery-worker-cpu-5d47cffb98-n86dz] Traceback (most recent call last):
[r-celery-worker-cpu-5d47cffb98-n86dz] File "/usr/local/lib/python3.6/site-packages/celery/backends/redis.py", line 399, in on_chord_part_return
[r-celery-worker-cpu-5d47cffb98-n86dz] callback.delay([unpack(tup, decode) for tup in resl])
[r-celery-worker-cpu-5d47cffb98-n86dz] File "/usr/local/lib/python3.6/site-packages/celery/backends/redis.py", line 399, in <listcomp>
[r-celery-worker-cpu-5d47cffb98-n86dz] callback.delay([unpack(tup, decode) for tup in resl])
[r-celery-worker-cpu-5d47cffb98-n86dz] File "/usr/local/lib/python3.6/site-packages/celery/backends/redis.py", line 350, in _unpack_chord_result
[r-celery-worker-cpu-5d47cffb98-n86dz] retval = self.exception_to_python(retval)
[r-celery-worker-cpu-5d47cffb98-n86dz] File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 281, in exception_to_python
[r-celery-worker-cpu-5d47cffb98-n86dz] exc = cls(*exc_msg if isinstance(exc_msg, tuple) else exc_msg)
[r-celery-worker-cpu-5d47cffb98-n86dz] TypeError: __init__() takes 1 positional argument but 2 were given

Expected Behavior

the chord should fail safely and call the on_error callback

Actual Behavior

the original error is not written, and on error cb not called

Issue 2:
Checklist

Not able to scale up Celery Beat using Docker containers running on multiple servers without replicating the tasks. E.g: If I have 2 containers running, it would run the same task twice.

Steps to reproduce

Expected behavior

Being able to scale up Celery Beat

Actual behavior