Repository: celery/celery

Issue:
celery worker no command line option for setting result backend
With celery 3.1.20 I get : 

```
(celery)alexv@asmodehn:~/Projects$ celery worker --help
Usage: celery worker [options] 

Start worker instance.

Examples::

    celery worker --app=proj -l info
    celery worker -A proj -l info -Q hipri,lopri

    celery worker -A proj --concurrency=4
    celery worker -A proj --concurrency=1000 -P eventlet

    celery worker --autoscale=10,0

Options:
  -A APP, --app=APP     app instance to use (e.g. module.attr_name)
  -b BROKER, --broker=BROKER
                        url to broker.  default is 'amqp://guest@localhost//'
  --loader=LOADER       name of custom loader class to use.
  --config=CONFIG       Name of the configuration module
  --workdir=WORKING_DIRECTORY
                        Optional directory to change to after detaching.
  -C, --no-color        
  -q, --quiet           
  -c CONCURRENCY, --concurrency=CONCURRENCY
                        Number of child processes processing the queue. The
                        default is the number of CPUs available on your
                        system.
  -P POOL_CLS, --pool=POOL_CLS
                        Pool implementation: prefork (default), eventlet,
                        gevent, solo or threads.
  --purge, --discard    Purges all waiting tasks before the daemon is started.
                        **WARNING**: This is unrecoverable, and the tasks will
                        be deleted from the messaging server.
  -l LOGLEVEL, --loglevel=LOGLEVEL
                        Logging level, choose between DEBUG, INFO, WARNING,
                        ERROR, CRITICAL, or FATAL.
  -n HOSTNAME, --hostname=HOSTNAME
                        Set custom hostname, e.g. 'w1.%h'. Expands: %h
                        (hostname), %n (name) and %d, (domain).
  -B, --beat            Also run the celery beat periodic task scheduler.
                        Please note that there must only be one instance of
                        this service.
  -s SCHEDULE_FILENAME, --schedule=SCHEDULE_FILENAME
                        Path to the schedule database if running with the -B
                        option. Defaults to celerybeat-schedule. The extension
                        ".db" may be appended to the filename. Apply
                        optimization profile.  Supported: default, fair
  --scheduler=SCHEDULER_CLS
                        Scheduler class to use. Default is
                        celery.beat.PersistentScheduler
  -S STATE_DB, --statedb=STATE_DB
                        Path to the state database. The extension '.db' may be
                        appended to the filename. Default: None
  -E, --events          Send events that can be captured by monitors like
                        celery events, celerymon, and others.
  --time-limit=TASK_TIME_LIMIT
                        Enables a hard time limit (in seconds int/float) for
                        tasks.
  --soft-time-limit=TASK_SOFT_TIME_LIMIT
                        Enables a soft time limit (in seconds int/float) for
                        tasks.
  --maxtasksperchild=MAX_TASKS_PER_CHILD
                        Maximum number of tasks a pool worker can execute
                        before it's terminated and replaced by a new worker.
  -Q QUEUES, --queues=QUEUES
                        List of queues to enable for this worker, separated by
                        comma. By default all configured queues are enabled.
                        Example: -Q video,image
  -X EXCLUDE_QUEUES, --exclude-queues=EXCLUDE_QUEUES
  -I INCLUDE, --include=INCLUDE
                        Comma separated list of additional modules to import.
                        Example: -I foo.tasks,bar.tasks
  --autoscale=AUTOSCALE
                        Enable autoscaling by providing max_concurrency,
                        min_concurrency. Example:: --autoscale=10,3 (always
                        keep 3 processes, but grow to 10 if necessary)
  --autoreload          Enable autoreloading.
  --no-execv            Don't do execv after multiprocessing child fork.
  --without-gossip      Do not subscribe to other workers events.
  --without-mingle      Do not synchronize with other workers at startup.
  --without-heartbeat   Do not send event heartbeats.
  --heartbeat-interval=HEARTBEAT_INTERVAL
                        Interval in seconds at which to send worker heartbeat
  -O OPTIMIZATION       
  -D, --detach          
  -f LOGFILE, --logfile=LOGFILE
                        Path to log file. If no logfile is specified, stderr
                        is used.
  --pidfile=PIDFILE     Optional file used to store the process pid. The
                        program will not start if this file already exists and
                        the pid is still alive.
  --uid=UID             User id, or user name of the user to run as after
                        detaching.
  --gid=GID             Group id, or group name of the main group to change to
                        after detaching.
  --umask=UMASK         Effective umask (in octal) of the process after
                        detaching.  Inherits the umask of the parent process
                        by default.
  --executable=EXECUTABLE
                        Executable to use for the detached process.
  --version             show program's version number and exit
  -h, --help            show this help message and exit
```

I do not see how I can set the result database.

In config file I have : 

```
BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = BROKER_URL
```

Then starting the worker I can pass `--broker=redis://another-host:6379`
But I cannot find anyway to change the result backend to override the configuration setting...

Am I missing something ?



Discussion:
In addition, it seems to me that the BROKER_URL value from the config module is not used when using `--config` command line option to start the worker... not sure if it s related or not ( or maybe just because I don't really get how configuration is working ?)

I have been trying to extend/specialize/wrap celery (https://github.com/asmodehn/celeros), but the way to configure it on runtime is still not clear to me...

OK I think I've pinned down the issue bothering me (beside the missing --result-backend option)...

I am using celery 3.1.20.

Here is a simplified configuration file to illustrate the issue : 

``` python
# config.py
CELERY_BROKER_URL = 'redis://localhost:6379/0'
BROKER_URL = CELERY_BROKER_URL
CELERY_RESULT_BACKEND = 'redis://localhost:6379/1'
```

To setup your app configuration you can do : 

``` python
# app.py
from . import config
from celery import Celery

my_app = Celery()
my_app.config_from_object(config)
```

and launch with : `python -m celery -A app`

XOR

``` python
# app.py
from . import config
from celery import Celery

my_app = Celery()
```

and launch with : `python -m celery -A app --config=config`

BUT NOT A MIX : 
In my case, doing something like this:

``` python
# app.py
from . import config
from celery import Celery

my_app = Celery()
my_app.config_from_object(config_default)
```

and launch with : `python -m celery -A app --config=config_override`
config_override content is ignored somehow (I noticed with BROKER_URL or CELERY_RESULT_BACKEND since they are very visible settings, celery worker display them on startup ) ...
Although I would have expected the values from config_override to replace the values configured from config_default...

Disclaimer : I haven't tested this code exactly though but if you cannot reproduce, let me know I'll dig more, when i get more time, to come up with a [SSCCE](http://www.sscce.org/)...

So for now I ll just drop the idea to have a "default"+"override" config strategy, in this way at least...

Same problem here: there is no way to dynamically set the backend of the worker.
Eg:
- I start a server where the celery app config is updated afterwards (custom redis port)
- I need the worker to know what the custom redis port is


Resolving patch:
diff --git a/celery/app/utils.py b/celery/app/utils.py
--- a/celery/app/utils.py
+++ b/celery/app/utils.py
@@ -102,6 +102,13 @@ def broker_url(self):
             self.first('broker_url', 'broker_host')
         )
 
+    @property
+    def result_backend(self):
+        return (
+            os.environ.get('CELERY_RESULT_BACKEND') or
+            self.get('CELERY_RESULT_BACKEND')
+        )
+
     @property
     def task_default_exchange(self):
         return self.first(
diff --git a/celery/bin/base.py b/celery/bin/base.py
--- a/celery/bin/base.py
+++ b/celery/bin/base.py
@@ -298,6 +298,7 @@ def add_preload_arguments(self, parser):
         group = parser.add_argument_group('Global Options')
         group.add_argument('-A', '--app', default=None)
         group.add_argument('-b', '--broker', default=None)
+        group.add_argument('--result-backend', default=None)
         group.add_argument('--loader', default=None)
         group.add_argument('--config', default=None)
         group.add_argument('--workdir', default=None)
@@ -467,6 +468,9 @@ def setup_app_from_commandline(self, argv):
         broker = preload_options.get('broker', None)
         if broker:
             os.environ['CELERY_BROKER_URL'] = broker
+        result_backend = preload_options.get('result_backend', None)
+        if result_backend:
+            os.environ['CELERY_RESULT_BACKEND'] = result_backend
         config = preload_options.get('config')
         if config:
             os.environ['CELERY_CONFIG_MODULE'] = config


Test patch:
diff --git a/t/unit/bin/test_base.py b/t/unit/bin/test_base.py
--- a/t/unit/bin/test_base.py
+++ b/t/unit/bin/test_base.py
@@ -166,6 +166,18 @@ def test_with_custom_broker(self, app):
             else:
                 os.environ.pop('CELERY_BROKER_URL', None)
 
+    def test_with_custom_result_backend(self, app):
+        prev = os.environ.pop('CELERY_RESULT_BACKEND', None)
+        try:
+            cmd = MockCommand(app=app)
+            cmd.setup_app_from_commandline(['--result-backend=xyzza://'])
+            assert os.environ.get('CELERY_RESULT_BACKEND') == 'xyzza://'
+        finally:
+            if prev:
+                os.environ['CELERY_RESULT_BACKEND'] = prev
+            else:
+                os.environ.pop('CELERY_RESULT_BACKEND', None)
+
     def test_with_custom_app(self, app):
         cmd = MockCommand(app=app)
         appstr = '.'.join([__name__, 'APP'])
@@ -276,8 +288,10 @@ def test_with_cmdline_config(self, app):
         cmd.namespace = 'worker'
         rest = cmd.setup_app_from_commandline(argv=[
             '--loglevel=INFO', '--',
+            'result.backend=redis://backend.example.com',
             'broker.url=amqp://broker.example.com',
             '.prefetch_multiplier=100'])
+        assert cmd.app.conf.result_backend == 'redis://backend.example.com'
         assert cmd.app.conf.broker_url == 'amqp://broker.example.com'
         assert cmd.app.conf.worker_prefetch_multiplier == 100
         assert rest == ['--loglevel=INFO']