Repository: celery/celery

Issue:
app.conf.worker_pool is not honored
The option `app.conf.worker_pool` is not honored. As I've been requested in previous issues to follow the template, I will try to.

# Checklist

- [X] I have verified that the issue exists against the `master` branch of Celery.
- [ ] This has already been asked to the [discussion group](https://groups.google.com/forum/#!forum/celery-users) first:  I didn't ask. This is a super easily reproducible bug, and I `git bisect` to identify the commit responsible.
- [X] I have read the relevant section in the
  [contribution guide](http://docs.celeryproject.org/en/latest/contributing.html#other-bugs)
  on reporting bugs.
- [X] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
- [X] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
- [X] I have checked the [commit log](https://github.com/celery/celery/commits/master)
  to find out if the bug was already fixed in the master branch.
- [X] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

## Mandatory Debugging Information

- [X] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [X] I have verified that the issue exists against the `master` branch of Celery.
- [X] I have included the contents of ``pip freeze`` in the issue.
- [X] I have included all the versions of all the external dependencies required
  to reproduce this bug.

#### Related Issues

- None

#### Possible Duplicates

- None

## Environment & Settings
<!-- Include the contents of celery --version below -->

5.0.5 (singularity)

<!-- Include the output of celery -A proj report below -->
<details>
<summary><b><code>celery report</code> Output:</b></summary>
<p>

```
software -> celery:5.0.5 (singularity) kombu:5.0.2 py:3.8.2
            billiard:3.6.3.0 py-amqp:5.0.5
platform -> system:Linux arch:64bit, ELF
            kernel version:4.19.121-linuxkit imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:disabled

deprecated_settings: None
worker_concurrency: 1
worker_pool: 'solo'
```

</p>
</details>

# Steps to Reproduce

## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->
* **Minimal Python Version**: N/A or Unknown
* **Minimal Celery Version**: N/A or Unknown
* **Minimal Kombu Version**: N/A or Unknown
* **Minimal Broker Version**: N/A or Unknown
* **Minimal Result Backend Version**: N/A or Unknown
* **Minimal OS and/or Kernel Version**: N/A or Unknown
* **Minimal Broker Client Version**: N/A or Unknown
* **Minimal Result Backend Client Version**: N/A or Unknown

### Python Packages
<!-- Please fill the contents of pip freeze below -->
<details>
<summary><b><code>pip freeze</code> Output:</b></summary>
<p>

```
amqp             5.0.5
billiard         3.6.3.0
celery           5.0.5   /tmp/celery
click            7.1.2
click-didyoumean 0.0.3
click-plugins    1.1.1
click-repl       0.1.6
kombu            5.0.2
pip              21.0.1
prompt-toolkit   3.0.18
pytz             2021.1
setuptools       54.1.2
six              1.15.0
vine             5.0.0
wcwidth          0.2.5
wheel            0.36.2
```

</p>
</details>

### Other Dependencies

<!--
Please provide system dependencies, configuration files
and other dependency information if applicable
-->
<details>
<p>
N/A
</p>
</details>

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.
-->

<details>
<p>

```python
from celery import Celery


app = Celery(
    'worker',
)

app.conf.update(
    worker_concurrency=1,
    worker_pool='solo',
)
```

</p>
</details>

# Expected Behavior
<!-- Describe in detail what you expect to happen -->

The banner displayed by `celery -A worker worker` should contain `concurrency: 1 (solo)`

# Actual Behavior
<!--
Describe in detail what actually happened.
Please include a backtrace and surround it with triple backticks (```).
In addition, include the Celery daemon logs, the broker logs,
the result backend logs and system logs below if they will help us debug
the issue.
-->

The banner contains `concurrency: 1 (prefork)`. The parameter `worker_concurrency` is honored, but `worker_pool` is ignored.

The problem apparently comes from 01651d2f5d9ad20dfb9812d92831510147974b23



Discussion:
Hey @brmzkw :wave:,
Thank you for opening an issue. We will get back to you as soon as we can.
Also, check out our [Open Collective](https://opencollective.com/celery) and consider backing us - every little helps!

We also offer priority support for our sponsors.
If you require immediate assistance please consider sponsoring us.


Resolving patch:
diff --git a/celery/app/base.py b/celery/app/base.py
--- a/celery/app/base.py
+++ b/celery/app/base.py
@@ -274,9 +274,10 @@ def __init__(self, main=None, loader=None, backend=None,
         self.__autoset('broker_url', broker)
         self.__autoset('result_backend', backend)
         self.__autoset('include', include)
-        self.__autoset('broker_use_ssl', kwargs.get('broker_use_ssl'))
-        self.__autoset('redis_backend_use_ssl',
-                       kwargs.get('redis_backend_use_ssl'))
+
+        for key, value in kwargs.items():
+            self.__autoset(key, value)
+
         self._conf = Settings(
             PendingConfiguration(
                 self._preconf, self._finalize_pending_conf),
diff --git a/celery/bin/worker.py b/celery/bin/worker.py
--- a/celery/bin/worker.py
+++ b/celery/bin/worker.py
@@ -45,8 +45,20 @@ def __init__(self):
     def convert(self, value, param, ctx):
         # Pools like eventlet/gevent needs to patch libs as early
         # as possible.
-        return concurrency.get_implementation(
-            value) or ctx.obj.app.conf.worker_pool
+        value = super().convert(value, param, ctx)
+        worker_pool = ctx.obj.app.conf.worker_pool
+        if value == 'prefork' and worker_pool:
+            # If we got the default pool through the CLI
+            # we need to check if the worker pool was configured.
+            # If the worker pool was configured, we shouldn't use the default.
+            value = concurrency.get_implementation(worker_pool)
+        else:
+            value = concurrency.get_implementation(value)
+
+            if not value:
+                value = concurrency.get_implementation(worker_pool)
+
+        return value
 
 
 class Hostname(StringParamType):
@@ -140,7 +152,8 @@ def detach(path, argv, logfile=None, pidfile=None, uid=None,
               '--statedb',
               cls=CeleryOption,
               type=click.Path(),
-              callback=lambda ctx, _, value: value or ctx.obj.app.conf.worker_state_db,
+              callback=lambda ctx, _,
+              value: value or ctx.obj.app.conf.worker_state_db,
               help_group="Worker Options",
               help="Path to the state database. The extension '.db' may be "
                    "appended to the filename.")
@@ -161,7 +174,8 @@ def detach(path, argv, logfile=None, pidfile=None, uid=None,
 @click.option('--prefetch-multiplier',
               type=int,
               metavar="<prefetch multiplier>",
-              callback=lambda ctx, _, value: value or ctx.obj.app.conf.worker_prefetch_multiplier,
+              callback=lambda ctx, _,
+              value: value or ctx.obj.app.conf.worker_prefetch_multiplier,
               cls=CeleryOption,
               help_group="Worker Options",
               help="Set custom prefetch multiplier value"
@@ -170,7 +184,8 @@ def detach(path, argv, logfile=None, pidfile=None, uid=None,
               '--concurrency',
               type=int,
               metavar="<concurrency>",
-              callback=lambda ctx, _, value: value or ctx.obj.app.conf.worker_concurrency,
+              callback=lambda ctx, _,
+              value: value or ctx.obj.app.conf.worker_concurrency,
               cls=CeleryOption,
               help_group="Pool Options",
               help="Number of child processes processing the queue.  "
@@ -268,7 +283,8 @@ def detach(path, argv, logfile=None, pidfile=None, uid=None,
 @click.option('-s',
               '--schedule-filename',
               '--schedule',
-              callback=lambda ctx, _, value: value or ctx.obj.app.conf.beat_schedule_filename,
+              callback=lambda ctx, _,
+              value: value or ctx.obj.app.conf.beat_schedule_filename,
               cls=CeleryOption,
               help_group="Embedded Beat Options")
 @click.option('--scheduler',


Test patch:
diff --git a/t/unit/app/test_app.py b/t/unit/app/test_app.py
--- a/t/unit/app/test_app.py
+++ b/t/unit/app/test_app.py
@@ -274,6 +274,10 @@ def test_with_broker(self, patching):
         with self.Celery(broker='foo://baribaz') as app:
             assert app.conf.broker_url == 'foo://baribaz'
 
+    def test_pending_confugration__kwargs(self):
+        with self.Celery(foo='bar') as app:
+            assert app.conf.foo == 'bar'
+
     def test_pending_configuration__setattr(self):
         with self.Celery(broker='foo://bar') as app:
             app.conf.task_default_delivery_mode = 44