Issue 1:
Checklist

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

#3772

Possible Duplicates

None

Environment & Settings

Celery version: celery:4.3.0 (rhubarb)

celery report Output:

software -> celery:4.3.0 (rhubarb) kombu:4.6.1 py:3.7.3
billiard:3.6.0.0 redis:3.2.1
platform -> system:Darwin arch:64bit
kernel version:18.6.0 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:disabled

BROKER_URL: 'redis://localhost:6379/0'
CELERY_DEFAULT_QUEUE: 'test'

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown

Minimal Celery Version: N/A or Unknown

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

amqp==2.5.0
billiard==3.6.0.0
celery==4.3.0
kombu==4.6.1
pytz==2019.1
redis==3.2.1
vine==1.3.0

Other Dependencies

N/A

Minimally Reproducible Test Case

from celery import Celery

class BaseConfig:
BROKER_URL = 'redis://localhost:6379/0'
CELERY_TASK_DEFAULT_QUEUE = 'test'

celery_app = Celery()
celery_app.config_from_object(BaseConfig)

@celery_app.task
def simple_task():
raise Exception("Testing Celery exception")

Then run with cmd:

celery worker -A app:celery_app -l debug -f celery.log

Expected Behavior

I expect the routing key=test and exchange=test.

Actual Behavior

The configuration CELERY_TASK_DEFAULT_QUEUE = 'test' is not picked up and defaults are used instead:

-------------- celery@Olegs-MBP.homenet.telecomitalia.it v4.3.0 (rhubarb)
---- **** -----
--- * *** * -- Darwin-18.6.0-x86_64-i386-64bit 2019-06-09 01:15:55
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: __main__:0x10d316518
- ** ---------- .> transport: redis://localhost:6379/0
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery

[tasks]
. app.simple_task
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap

Dropping the TASK_ from the config property name to CELERY_DEFAULT_QUEUE = 'test' works as intended:

-------------- celery@Olegs-MBP.homenet.telecomitalia.it v4.3.0 (rhubarb)
---- **** -----
--- * *** * -- Darwin-18.6.0-x86_64-i386-64bit 2019-06-09 01:19:19
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: __main__:0x1085be2b0
- ** ---------- .> transport: redis://localhost:6379/0
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> test exchange=test(direct) key=test

[tasks]
. app.simple_task
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap

Now, issue #3772 was reporting incorrect documentation, i.e. CELERY_DEFAULT_QUEUE should be CELERY_TASK_DEFAULT_QUEUE and so all old configs without the TASK_ have been updated in #4094 to have it. However, since the fix to the docs, the code actually regressed and now accepts the older settings.

Issue 2:
Checklist

[ x ] I have checked the issues list

for similar or identical enhancement to an existing feature.

[ x ] I have checked the pull requests list

for existing proposed enhancements.

[ x ] I have checked the commit log

to find out if the if the same enhancement was already implemented in the

main branch.

[ x ] I have included all related issues and possible duplicate issues in this issue

(If there are none, check this box anyway).

Related Issues and Possible Duplicates

Related Issues

sontek/pyramid_celery#101

Possible Duplicates

None

Brief Summary

The pyramid-celery package adds ini and ini-var options to the standard commands. These allow the standard Pyramid ini configuration files to config Celery. With current Celery however, only the worker, beat, and events are able to be hooked into pyramid-celery's approach because those commands are set with.

context_settings={
'allow_extra_args': True
}

See beat for example:

celery/celery/bin/beat.py

Line 11
in
f3a2cf4

'allow_extra_args': True

The shell and purge commands for some reason do not have context_settings and so do not set allow_extra_args. See for example in shell, https://github.com/celery/celery/blob/f3a2cf45a69b443cac6c79a5c85583c8bd91b0a3/celery/bin/shell.py#LL82C1-L117C24

The consequence is that the ini command options cause errors since these are extra and unknown args to these commands. Excluding the ini config casues shell when it comes up to not get the task registration configs and purge isn't able to see queues.

Design

Architectural Considerations

pyramid-celery is on pypi. Allows you to use pyramid .ini files to configure celery and have your pyramid configuration inside celery tasks.

Proposed Behavior

This should not affect Celery functionality, but will allow pyramid-celery to function as expected. See related issue tracked in pyramid-celery here sontek/pyramid_celery#101

Proposed UI/UX

Celery should add context_settings with allow_extra_args,

context_settings={
'allow_extra_args': True
}

to both the shell and purge commands. This will allow projects like pyramid-celery to hook in their own custom configuration options using the recommended (by Celery) celery_app.user_options['preload'].add() and/or celery_app.user_options[option].add() approaches. See here. This will also make shell and purge commands consistent with other commands like worker, beat, and events that already have these settings.

Diagrams

N/A

Alternatives

None