Issue 1:
My use case: I have Macbook Pro M1 laptop as my host machine. I want to run celery worker nodes inside linux based docker containers.

Ubuntu 18.04 as host machine works without any problem, but not the macbook pro with M1 processor. I can create tasks from my host machine, but dockerized workers does not work on anything.

Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version: 5.0.5

celery report Output:

### host machine
software -> celery:5.0.5 (singularity) kombu:5.0.2 py:3.9.2
billiard:3.6.3.0 redis:3.5.3
platform -> system:Darwin arch:64bit
kernel version:20.3.0 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://localhost:6379/0

broker_url: 'redis://localhost:6379/0'
result_backend: 'redis://localhost:6379/0'
deprecated_settings: None

### inside docker
software -> celery:5.0.5 (singularity) kombu:5.0.2 py:3.9.2
billiard:3.6.3.0 redis:3.5.3
platform -> system:Linux arch:64bit, ELF
kernel version:4.19.121-linuxkit imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://host.docker.internal:6379/0

broker_url: 'redis://host.docker.internal:6379/0'
result_backend: 'redis://host.docker.internal:6379/0'
deprecated_settings: None

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.7 or 3.9

Minimal Celery Version: 5.0.5

Minimal Kombu Version: see below

Minimal Broker Version: see below

Minimal Result Backend Version: see below

Minimal OS and/or Kernel Version: see below

Minimal Broker Client Version: see below

Minimal Result Backend Client Version: see below

Python Packages

pip freeze Output:

### host machine
amqp==5.0.5
billiard==3.6.3.0
celery @ https://github.com/celery/celery/zipball/master
click==7.1.2
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.1.6
kombu==5.0.2
prompt-toolkit==3.0.18
pytz==2021.1
redis==3.5.3
six==1.15.0
vine==5.0.0
wcwidth==0.2.5

### inside docker
amqp==5.0.5
billiard @ https://github.com/celery/billiard/zipball/master
celery @ https://github.com/celery/celery/zipball/master
click==7.1.2
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.1.6
kombu @ https://github.com/celery/kombu/zipball/master
prompt-toolkit==3.0.18
pytz==2021.1
redis==3.5.3
six==1.15.0
vine @ https://github.com/celery/vine/zipball/master
wcwidth==0.2.5

Other Dependencies

### host machine
% uname -a
Darwin Ignacios-MBP.fritz.box 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan 21 00:06:51 PST 2021; root:xnu-7195.81.3~1/RELEASE_ARM64_T8101 arm64

### inside docker
% uname -a
Linux 29461f0a74a4 4.19.121-linuxkit #1 SMP PREEMPT Thu Jan 21 15:45:22 UTC 2021 aarch64 GNU/Linux

for redis and docker see test case 0. below

Minimally Reproducible Test Case

Host machine setup environment: using redis and docker

% redis-server --version
Redis server v=6.2.1 sha=00000000:0 malloc=libc bits=64 build=cfaa1431404ef25b

% docker version
Client: Docker Engine - Community
Version: 20.10.5
API version: 1.41
Go version: go1.16
Git commit: 55c4c88
Built: Wed Mar 3 00:56:04 2021
OS/Arch: darwin/arm64
Context: default
Experimental: true

Server: Docker Engine - Community
Engine:
Version: 20.10.5
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 363e9a8
Built: Tue Mar 2 20:16:48 2021
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.4.4
GitCommit: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
runc:
Version: 1.0.0-rc93
GitCommit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
docker-init:
Version: 0.19.0
GitCommit: de40ad0

Run redis-server on host machine

redis-server --protected mode no --loglevel verbose

Create a dockerized worker, run it

# tasks.py
from celery import Celery
host = "host.docker.internal"
redis_str = f'redis://{host}:6379/0'
app = Celery('tasks', broker=redis_str, backend=redis_str)
@app.task
def add(x, y):
return x + y

# Dockerfile
FROM python:3.9
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8 PYTHONUNBUFFERED=1

RUN pip install https://github.com/celery/celery/zipball/master#egg=celery \
https://github.com/celery/billiard/zipball/master#egg=billiard \
https://github.com/celery/kombu/zipball/master#egg=kombu \
https://github.com/celery/vine/zipball/master#egg=vine
RUN pip install redis

COPY . /app
WORKDIR /app
CMD ["celery", "-A", "tasks", "worker", "-l" ,"DEBUG"]

docker build -f Dockerfile -t celery_dockerized .
docker run celery_dockerized

Setup host machine celery, change host="localhost" for the host machine in tasks.py

python3 -m venv venv # tried with python 3.7 and 3.9
source venv/bin/activate
pip install https://github.com/celery/celery/zipball/master#egg=celery \
https://github.com/celery/billiard/zipball/master#egg=billiard \
https://github.com/celery/kombu/zipball/master#egg=kombu \
https://github.com/celery/vine/zipball/master#egg=vine
pip install redis

# tasks.py
from celery import Celery
host = "localhost"
redis_str = f'redis://{host}:6379/0'
app = Celery('tasks', broker=redis_str, backend=redis_str)
@app.task
def add(x, y):
return x + y

Trigger the task from host machine, it will give the task id

% celery -A tasks call --args='[1,2]' tasks.add

<some_task_id>

Expected Behavior

Dockerized containers (linux) take and execute tasks that are initiated from host machine (mac m1).

Actual Behavior

Dockerized containers (linux) does not take any tasks that are initiated from host machine (mac m1). Also, they also cannot see any other worker nodes on host machine. But, they can see other worker nodes within the docker container. There are no error/warning messages on celery workers.

Redis server logs, when a dockerized worker created

18405:M 24 Mar 2021 17:13:05.576 - DB 0: 2 keys (0 volatile) in 8 slots HT.
18405:M 24 Mar 2021 17:13:09.565 - Accepted 192.168.64.6:57300
18405:M 24 Mar 2021 17:13:09.566 - Accepted 192.168.64.6:57302
18405:M 24 Mar 2021 17:13:10.740 - DB 0: 2 keys (0 volatile) in 8 slots HT.
18405:M 24 Mar 2021 17:13:14.581 - Accepted 192.168.64.6:57304
18405:M 24 Mar 2021 17:13:14.615 - Accepted 192.168.64.6:57306
18405:M 24 Mar 2021 17:13:14.619 - Accepted 192.168.64.6:57308
18405:M 24 Mar 2021 17:13:15.648 - Accepted 192.168.64.6:57310
18405:M 24 Mar 2021 17:13:15.654 - Accepted 192.168.64.6:57312
18405:M 24 Mar 2021 17:13:15.663 - Accepted 192.168.64.6:57314
18405:M 24 Mar 2021 17:13:15.669 - Accepted 192.168.64.6:57316
18405:M 24 Mar 2021 17:13:15.678 - Accepted 192.168.64.6:57318
18405:M 24 Mar 2021 17:13:15.717 * 100 changes in 300 seconds. Saving...

Worker node running inside docker, when a task created from host machine nothing happens. Or, when there is a new worker on host machine also nothing..

[2021-03-24 15:29:12,402: INFO/MainProcess] Connected to redis://host.docker.internal:6379/0
[2021-03-24 15:29:12,402: DEBUG/MainProcess] ^-- substep ok
[2021-03-24 15:29:12,403: DEBUG/MainProcess] | Consumer: Starting Events
[2021-03-24 15:29:12,424: DEBUG/MainProcess] ^-- substep ok
[2021-03-24 15:29:12,424: DEBUG/MainProcess] | Consumer: Starting Mingle
[2021-03-24 15:29:12,425: INFO/MainProcess] mingle: searching for neighbors
[2021-03-24 15:29:13,463: INFO/MainProcess] mingle: all alone
[2021-03-24 15:29:13,463: DEBUG/MainProcess] ^-- substep ok
[2021-03-24 15:29:13,463: DEBUG/MainProcess] | Consumer: Starting Gossip
[2021-03-24 15:29:13,477: DEBUG/MainProcess] ^-- substep ok
[2021-03-24 15:29:13,478: DEBUG/MainProcess] | Consumer: Starting Tasks
[2021-03-24 15:29:13,485: DEBUG/MainProcess] ^-- substep ok
[2021-03-24 15:29:13,485: DEBUG/MainProcess] | Consumer: Starting Control
[2021-03-24 15:29:13,494: DEBUG/MainProcess] ^-- substep ok
[2021-03-24 15:29:13,494: DEBUG/MainProcess] | Consumer: Starting Heart
[2021-03-24 15:29:13,500: DEBUG/MainProcess] ^-- substep ok
[2021-03-24 15:29:13,501: DEBUG/MainProcess] | Consumer: Starting event loop
[2021-03-24 15:29:13,501: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2021-03-24 15:29:13,501: INFO/MainProcess] celery@d03f98ad699a ready.

Same worker node above, when another one created in the same docker container, or inside a new container. Also, it runs tasks that are triggered inside the same or other container just fine.

[2021-03-24 15:45:40,711: DEBUG/MainProcess] pidbox received method hello(from_node='celery@10dde1c3db69', revoked={}) [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'acf00853-a0b7-3d14-bb4d-92295b74b1b8'} ticket:3bc055e9-d508-4f18-add5-bf80ffb83aa9]
[2021-03-24 15:45:40,711: INFO/MainProcess] sync with celery@10dde1c3db69
[2021-03-24 15:45:41,765: DEBUG/MainProcess] celery@10dde1c3db69 joined the party

Issue 2:
Checklist

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

#4777 (but the work around "delete celerybeat-schedule file" does not work)

Environment & Settings

Celery version:

4.3.0 (rhubarb)

celery report Output:

software -> celery:4.3.0 (rhubarb) kombu:4.4.0 py:3.7.2+
billiard:3.6.0.0 redis:3.2.1
platform -> system:Linux arch:64bit, ELF
kernel version:4.19.0-4-amd64 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://localhost:6379/0

BROKER_URL: 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND: 'redis://localhost:6379/0'
ENV: 'production'
DEBUG: False
TESTING: False
PROPAGATE_EXCEPTIONS: None
PRESERVE_CONTEXT_ON_EXCEPTION: None
SECRET_KEY: '********'
PERMANENT_SESSION_LIFETIME: datetime.timedelta(days=31)
USE_X_SENDFILE: False
SERVER_NAME: None
APPLICATION_ROOT: '/'
SESSION_COOKIE_NAME: 'session'
SESSION_COOKIE_DOMAIN: None
SESSION_COOKIE_PATH: None
SESSION_COOKIE_HTTPONLY: True
SESSION_COOKIE_SECURE: False
SESSION_COOKIE_SAMESITE: None
SESSION_REFRESH_EACH_REQUEST: True
MAX_CONTENT_LENGTH: None
SEND_FILE_MAX_AGE_DEFAULT: datetime.timedelta(seconds=43200)
TRAP_BAD_REQUEST_ERRORS: None
TRAP_HTTP_EXCEPTIONS: False
EXPLAIN_TEMPLATE_LOADING: False
PREFERRED_URL_SCHEME: 'http'
JSON_AS_ASCII: True
JSON_SORT_KEYS: '********'
JSONIFY_PRETTYPRINT_REGULAR: False
JSONIFY_MIMETYPE: 'application/json'
TEMPLATES_AUTO_RELOAD: None
MAX_COOKIE_SIZE: 4093
CACHE_DEFAULT_TIMEOUT: 300
CACHE_TYPE: 'redis'
CELERYBEAT_SCHEDULE: {
'purge_old_data': { 'kwargs': {'max_hours': 48},
'schedule': datetime.timedelta(seconds=600),
'task': 'purge_old_data'},
'update-country-codes': { 'schedule': datetime.timedelta(seconds=60),
'task': 'update_receivers_country_code'},
'update-ddb': { 'schedule': datetime.timedelta(seconds=3600),
'task': 'import_ddb'},
'update-logbook': { 'schedule': datetime.timedelta(seconds=60),
'task': 'update_logbook_entries'},
'update-max-altitudes': { 'schedule': <crontab: 30 0 * * * (m/h/d/dM/MY)>,
'task': 'update_logbook_max_altitude'},
'update-stats-daily': { 'kwargs': {'day_offset': -1},
'schedule': <crontab: 5 0 * * * (m/h/d/dM/MY)>,
'task': 'update_stats'},
'update-takeoff-and-landing': { 'kwargs': {'last_minutes': 10},
'schedule': datetime.timedelta(seconds=60),
'task': 'update_takeoff_landings'}}
CELERY_BROKER_URL: 'redis://localhost:6379/0'
SQLALCHEMY_DATABASE_URI: '********'
SQLALCHEMY_TRACK_MODIFICATIONS: False
BOOTSTRAP_USE_MINIFIED: True
BOOTSTRAP_CDN_FORCE_SSL: False
BOOTSTRAP_QUERYSTRING_REVVING: True
BOOTSTRAP_SERVE_LOCAL: False
BOOTSTRAP_LOCAL_SUBDOMAIN: None
SQLALCHEMY_BINDS: None
SQLALCHEMY_NATIVE_UNICODE: None
SQLALCHEMY_ECHO: False
SQLALCHEMY_RECORD_QUERIES: None
SQLALCHEMY_POOL_SIZE: None
SQLALCHEMY_POOL_TIMEOUT: None
SQLALCHEMY_POOL_RECYCLE: None
SQLALCHEMY_MAX_OVERFLOW: None
SQLALCHEMY_COMMIT_ON_TEARDOWN: False

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.7

Minimal Celery Version: N/A or Unknown

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

aerofiles==0.4.1
alembic==1.0.8
amqp==2.4.2
Babel==2.6.0
billiard==3.6.0.0
celery==4.3.0
certifi==2019.3.9
chardet==3.0.4
Click==7.0
coverage==4.5.3
coveralls==1.2.0
docopt==0.6.2
dominate==2.3.5
flake8==3.5.0
Flask==1.0.2
Flask-Bootstrap==3.3.7.1
Flask-Caching==1.7.0
Flask-Migrate==2.3.1
flask-nav==0.6
Flask-SQLAlchemy==2.3.2
flower==0.9.2
GeoAlchemy2==0.5.0
geographiclib==1.49
geopy==1.17.0
idna==2.8
itsdangerous==1.1.0
Jinja2==2.10
kombu==4.4.0
lxml==4.3.2
Mako==1.0.7
MarkupSafe==1.1.1
mccabe==0.6.1
mgrs==1.3.5
nose==1.3.7
ogn-client==0.9.1
-e git+http://github.com/Meisterschueler/ogn-python@a177049d6e6d02e288b6ed141bab04efd878e53f#egg=ogn_python
pkg-resources==0.0.0
psycopg2-binary==2.7.6.1
pycodestyle==2.3.1
pyflakes==1.6.0
python-dateutil==2.8.0
python-editor==1.0.4
pytz==2018.9
redis==3.2.1
requests==2.21.0
Shapely==1.5.17
six==1.12.0
SQLAlchemy==1.3.1
tornado==5.1.1
tqdm==4.28.1
urllib3==1.24.1
vine==1.3.0
visitor==0.1.3
Werkzeug==0.14.1
xmlunittest==0.5.0

Other Dependencies

N/A

Minimally Reproducible Test Case

start the redis server redis-server

start the worker celery -A ogn_python.collect worker -l info

start the beat celery -A ogn_python.collect beat -l info

Expected Behavior

Beat is running

Actual Behavior

Beat is crashing

$ celery -A ogn_python.collect beat -l info
celery beat v4.3.0 (rhubarb) is starting.
__ - ... __ - _
LocalTime -> 2019-04-08 20:59:02
Configuration ->
. broker -> redis://localhost:6379/0
. loader -> celery.loaders.app.AppLoader
. scheduler -> celery.beat.PersistentScheduler
. db -> celerybeat-schedule
. logfile -> [stderr]@%INFO
. maxinterval -> 5.00 minutes (300s)
[2019-04-08 20:59:02,200: INFO/MainProcess] beat: Starting...
[2019-04-08 20:59:02,216: CRITICAL/MainProcess] beat raised exception <class '_dbm.error'>: error('cannot add item to database')
Traceback (most recent call last):
File "/var/www/html/ogn-python/venv/lib/python3.7/site-packages/kombu/utils/objects.py", line 42, in __get__
return obj.__dict__[self.__name__]
KeyError: 'scheduler'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.7/shelve.py", line 111, in __getitem__
value = self.cache[key]
KeyError: 'entries'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/www/html/ogn-python/venv/lib/python3.7/site-packages/celery/beat.py", line 524, in _create_schedule
self._store[str('entries')]
File "/usr/lib/python3.7/shelve.py", line 113, in __getitem__
f = BytesIO(self.dict[key.encode(self.keyencoding)])
KeyError: b'entries'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/www/html/ogn-python/venv/lib/python3.7/site-packages/celery/apps/beat.py", line 109, in start_scheduler
service.start()
File "/var/www/html/ogn-python/venv/lib/python3.7/site-packages/celery/beat.py", line 588, in start
humanize_seconds(self.scheduler.max_interval))
File "/var/www/html/ogn-python/venv/lib/python3.7/site-packages/kombu/utils/objects.py", line 44, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "/var/www/html/ogn-python/venv/lib/python3.7/site-packages/celery/beat.py", line 632, in scheduler
return self.get_scheduler()
File "/var/www/html/ogn-python/venv/lib/python3.7/site-packages/celery/beat.py", line 627, in get_scheduler
lazy=lazy,
File "/var/www/html/ogn-python/venv/lib/python3.7/site-packages/celery/beat.py", line 467, in __init__
Scheduler.__init__(self, *args, **kwargs)
File "/var/www/html/ogn-python/venv/lib/python3.7/site-packages/celery/beat.py", line 226, in __init__
self.setup_schedule()
File "/var/www/html/ogn-python/venv/lib/python3.7/site-packages/celery/beat.py", line 495, in setup_schedule
self._create_schedule()
File "/var/www/html/ogn-python/venv/lib/python3.7/site-packages/celery/beat.py", line 528, in _create_schedule
self._store[str('entries')] = {}
File "/usr/lib/python3.7/shelve.py", line 125, in __setitem__
self.dict[key.encode(self.keyencoding)] = f.getvalue()
_dbm.error: cannot add item to database