Issue 1:
I am running celery with RabbitMQ as my backend, using a postgresql database as the result backend.

I have created a task that I want to wait on:

testing/failure/tasks.py:

from celery.decorators import task

@task
def return_result():
return "Should return"

I call this task in a separate client process as follows:

testing/client.py:

if __name__ == '__main__':
from failure import tasks

returned_results = tasks.return_result.apply_async(immediate=True).get()
print "Returned results: %s" % returned_results

This call blocks on the .get() call and never returns the result.

testing/celeryconfig.py:

# List of modules to import when celery starts.
CELERY_IMPORTS = (
"failure.tasks",
)

## Result store settings.
CELERY_RESULT_BACKEND = "database"
CELERY_RESULT_DBURI = 'postgresql://user:password@db_host:5432/crose_saas_surveysystem'

# override for testing
#CELERY_RESULT_DBURI = "sqlite:////tmp/celery.sqlite"

CELERY_DEFAULT_QUEUE = "default"
CELERY_QUEUES = {
"default": {
"binding_key": "task.#",
},
}
CELERY_DEFAULT_EXCHANGE = "tasks"
CELERY_DEFAULT_EXCHANGE_TYPE = "topic"
CELERY_DEFAULT_ROUTING_KEY = "task.default"

CELERY_ROUTES = {
}

## Broker settings.
BROKER_HOST = 'localhost'
BROKER_PORT = 5672
BROKER_VHOST = "/"
BROKER_USER = "guest"
BROKER_PASSWORD = "guest"

## Worker settings
## If you're doing mostly I/O you can have more processes,
## but if mostly spending CPU, try to keep it close to the
## number of CPUs on your machine. If not set, the number of CPUs/cores
## available will be used.
CELERYD_CONCURRENCY = 10
# CELERYD_LOG_FILE = "celeryd.log"
CELERYD_LOG_LEVEL = "DEBUG"

I'll attach the log file in a subsequent comment

Issue 2:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version:

5.0.1 (singularity)

celery report Output:

software -> celery:5.0.1 (singularity) kombu:5.0.2 py:3.7.3
billiard:3.6.3.0 redis:3.5.3
platform -> system:Linux arch:32bit, ELF
kernel version:5.4.72+ imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://localhost:6379/0

BROKER_URL: 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND: 'redis://localhost:6379/0'
deprecated_settings: None
ENV: 'production'
DEBUG: False
TESTING: False
PROPAGATE_EXCEPTIONS: None
PRESERVE_CONTEXT_ON_EXCEPTION: None
SECRET_KEY: '********'
PERMANENT_SESSION_LIFETIME: datetime.timedelta(days=31)
USE_X_SENDFILE: False
SERVER_NAME: None
APPLICATION_ROOT: '/'
SESSION_COOKIE_NAME: 'session'
SESSION_COOKIE_DOMAIN: None
SESSION_COOKIE_PATH: None
SESSION_COOKIE_HTTPONLY: True
SESSION_COOKIE_SECURE: False
SESSION_COOKIE_SAMESITE: None
SESSION_REFRESH_EACH_REQUEST: True
MAX_CONTENT_LENGTH: None
SEND_FILE_MAX_AGE_DEFAULT: datetime.timedelta(seconds=43200)
TRAP_BAD_REQUEST_ERRORS: None
TRAP_HTTP_EXCEPTIONS: False
EXPLAIN_TEMPLATE_LOADING: False
PREFERRED_URL_SCHEME: 'http'
JSON_AS_ASCII: True
JSON_SORT_KEYS: '********'
JSONIFY_PRETTYPRINT_REGULAR: False
JSONIFY_MIMETYPE: 'application/json'
TEMPLATES_AUTO_RELOAD: None
MAX_COOKIE_SIZE: 4093
CELERY_BROKER_URL: 'redis://localhost:6379/0'
CELERY_TASK_TRACK_STARTED: True

Steps to Reproduce

Required Dependencies

Minimal Python Version: Python 3.7.3

Minimal Celery Version: 5.0.1 (singularity)

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: redis==3.5.3

Minimal Result Backend Version: redis==3.5.3

Minimal OS and/or Kernel Version: Raspbian GNU/Linux 10 (buster)

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

amqp==5.0.1
asn1crypto==0.24.0
billiard==3.6.3.0
cachelib==0.1.1
celery==5.0.1
certifi==2018.8.24
chardet==3.0.4
click==7.1.2
click-didyoumean==0.0.3
click-repl==0.1.6
configparser==5.0.1
cryptography==2.6.1
defusedxml==0.6.0
dominate==2.6.0
entrypoints==0.3
ExifReader==0.1.1
Flask==1.1.2
Flask-Bootstrap==3.3.7.1
Flask-Login==0.5.0
gphoto2==2.2.3
gunicorn==20.0.4
idna==2.6
importlib-metadata==2.0.0
itsdangerous==1.1.0
Jinja2==2.11.2
keyring==17.1.1
keyrings.alt==3.1.1
kombu==5.0.2
MarkupSafe==1.1.1
Pillow==8.0.1
prompt-toolkit==3.0.8
psutil==5.7.3
pycrypto==2.6.1
PyGObject==3.30.4
python-apt==1.8.4.1
pytz==2020.1
pyxdg==0.25
redis==3.5.3
requests==2.21.0
SecretStorage==2.3.1
six==1.12.0
smbus2==0.3.0
ssh-import-id==5.7
urllib3==1.24.1
vine==5.0.0
visitor==0.1.3
wcwidth==0.2.5
Werkzeug==1.0.1
zipp==3.4.0

Other Dependencies

N/A

Minimally Reproducible Test Case

I'm adding Celery to my "intvlm8r" project running on a Pi, and my dev build against 4.4.7 runs great.

All my v5 builds fail. I'm daemonising celery using systemd, and all attempts to start the service fail with "Error: no such option: --logfile".

The celery files are all in this branch of my repo, all in the root of the RaspberryPi folder. (My install script moves them to their correct locations).

I've tried tweaking the config to get it to run, and it was only when I realised I was running 5.0.1 on one and 4.4.7 on the dev Pi that I did the comparison.

I'm installing Celery with sudo pip3 install "celery[redis]", and when that pulls 5.0.1 we fail.

Re-image the card with Buster Lite, update, re-install with 'sudo pip3 install "celery[redis]"==4.4.7` it comes up cleanly every time.

celery.conf

d /var/run/celery 0755 pi www-data -
d /var/log/celery 0755 pi www-data -

celery

ENABLED="true"
CELERYD_NODES="celery_worker"

# Where to chdir at start.
CELERYD_CHDIR="/home/pi/www"

CELERYD_OPTS="--time-limit=300 --concurrency=8"

CELERY_APP="intvlm8r.celery"

# %n will be replaced with the nodename.
CELERYD_LOG_FILE="/var/log/celery/%N.log"
CELERYD_PID_FILE="/var/run/celery/%N.pid"
CELERYD_LOG_LEVEL="INFO"

# Workers should run as an unprivileged user.
CELERYD_USER="pi"
CELERYD_GROUP="www-data"

CELERY_CREATE_DIRS=1

CELERY_BIN="/usr/local/bin/celery"

celery.service

[Unit]
Description=Celery Service
Requires=intvlm8r.service
After=network.target intvlm8r.service

[Service]
Type=forking
User=pi
Group=www-data
EnvironmentFile=/etc/default/celery
WorkingDirectory=/home/pi/www

ExecStart=/bin/sh -c '${CELERY_BIN} -A ${CELERY_APP} --logfile=${CELERYD_LOG_FILE} \
--loglevel=${CELERYD_LOG_LEVEL} multi start ${CELERYD_NODES} ${CELERYD_OPTS}'

ExecStop=/bin/sh -c '${CELERY_BIN} stopwait ${CELERYD_NODES}'

ExecReload=/bin/sh -c '${CELERY_BIN} -A ${CELERY_APP} restart ${CELERYD_NODES} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'

[Install]
WantedBy=multi-user.target

Expected Behavior

Celery will start without complaint.

Actual Behavior

All attempts to start the service fail with "Error: no such option: --logfile".

pi@ThursPiRC:~ $ sudo systemctl status celery
● celery.service - Celery Service
Loaded: loaded (/etc/systemd/system/celery.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2020-10-30 17:03:45 AEDT; 18h ago
Process: 402 ExecStart=/bin/sh -c ${CELERY_BIN} -A ${CELERY_APP} --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} multi start ${CELERYD_NODES} ${CELERYD_OPTS} (c

Oct 30 17:03:23 ThursPiRC systemd[1]: Starting Celery Service...
Oct 30 17:03:44 ThursPiRC sh[402]: Usage: celery [OPTIONS] COMMAND [ARGS]...
Oct 30 17:03:44 ThursPiRC sh[402]: Try 'celery --help' for help.
Oct 30 17:03:44 ThursPiRC sh[402]: Error: no such option: --logfile
Oct 30 17:03:45 ThursPiRC systemd[1]: celery.service: Control process exited, code=exited, status=2/INVALIDARGUMENT
Oct 30 17:03:45 ThursPiRC systemd[1]: celery.service: Failed with result 'exit-code'.
Oct 30 17:03:45 ThursPiRC systemd[1]: Failed to start Celery Service.

journalctl -xn

-- Logs begin at Sat 2020-10-31 11:29:33 AEDT, end at Mon 2020-11-02 19:32:31 AEDT. --
Nov 02 19:32:26 ThursPiRC sh[675]: Usage: celery [OPTIONS] COMMAND [ARGS]...
Nov 02 19:32:26 ThursPiRC sh[675]: Try 'celery --help' for help.
Nov 02 19:32:26 ThursPiRC sh[675]: Error: no such option: --logfile
Nov 02 19:32:27 ThursPiRC systemd[1]: celery.service: Control process exited, code=exited, status=2/INVALIDARGUMENT
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An ExecStart= process belonging to unit celery.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 2.
Nov 02 19:32:27 ThursPiRC systemd[1]: celery.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit celery.service has entered the 'failed' state with result 'exit-code'.
Nov 02 19:32:27 ThursPiRC systemd[1]: Failed to start Celery Service.
-- Subject: A start job for unit celery.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit celery.service has finished with a failure.
--
-- The job identifier is 660 and the job result is failed.
Nov 02 19:32:27 ThursPiRC sudo[672]: pam_unix(sudo:session): session closed for user root
Nov 02 19:32:30 ThursPiRC sudo[679]: pi : TTY=pts/0 ; PWD=/home/pi/www ; USER=root ; COMMAND=/bin/systemctl status celery
Nov 02 19:32:30 ThursPiRC sudo[679]: pam_unix(sudo:session): session opened for user root by pi(uid=0)
Nov 02 19:32:31 ThursPiRC sudo[679]: pam_unix(sudo:session): session closed for user root

Please let me know if any other files will help.

Thanks,

- Greig.