Repository: celery/celery

Issue:
worker with concurrency running only a single task while multiple tasks reserved 
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
bug reports which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [ ] This has already been asked to the [discussion group](https://groups.google.com/forum/#!forum/celery-users) first.
- [x] I have read the relevant section in the
  [contribution guide](http://docs.celeryproject.org/en/latest/contributing.html#other-bugs)
  on reporting bugs.
- [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
- [x] I have checked the [commit log](https://github.com/celery/celery/commits/master)
  to find out if the bug was already fixed in the master branch.
- [x] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

## Mandatory Debugging Information

- [x] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [x] I have included the contents of ``pip freeze`` in the issue.
- [x] I have included all the versions of all the external dependencies required
  to reproduce this bug.

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->
- [ ] I have tried reproducing the issue on more than one Python version
  and/or implementation.
- [ ] I have tried reproducing the issue on more than one message broker and/or
  result backend.
- [ ] I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
- [ ] I have tried reproducing the issue on more than one operating system.
- [ ] I have tried reproducing the issue on more than one workers pool.
- [x] I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
- [x] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues

- None

#### Possible Duplicates

- None

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery Version**: 5.0.0 (singularity)
<!-- Include the output of celery -A proj report below -->
<details>
<summary><b><code>celery report</code> Output:</b></summary>
<p>

```
root@ad36e1cf0d81:/app# celery -A app.worker report

software -> celery:5.0.0 (singularity) kombu:5.0.2 py:3.7.5
            billiard:3.6.3.0 py-amqp:5.0.1
platform -> system:Linux arch:64bit
            kernel version:5.4.0-1025-aws imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:disabled

broker_url: 'amqp://prediction_celery:********@broker:5672//'
deprecated_settings: None
task_routes: {
 'app.worker.*': {'queue': 'high_memory_usage'}}
task_serializer: 'pickle'
accept_content: ['json', 'pickle']
broker_transport_options: {
    'interval_max': 0.5,
    'interval_start': 0,
    'interval_step': 0.2,
    'max_retries': 3}
worker_prefetch_multiplier: 1



software -> celery:5.0.0 (singularity) kombu:5.0.2 py:3.7.5
            billiard:3.6.3.0 py-amqp:5.0.1
platform -> system:Linux arch:64bit
            kernel version:5.4.0-1025-aws imp:CPython
loader   -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled

deprecated_settings: None
```

</p>
</details>

# Steps to Reproduce

## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->
* **Minimal Python Version**: 3.7.5
* **Minimal Celery Version**: 5.0.0
* **Minimal Kombu Version**: 5.0.2
* **Minimal Broker Version**: RabbitMQ version: 3.8.9
* **Minimal Result Backend Version**: N/A or Unknown
* **Minimal OS and/or Kernel Version**: N/A or Unknown
* **Minimal Broker Client Version**: N/A or Unknown
* **Minimal Result Backend Client Version**: N/A or Unknown

### Python Packages
<!-- Please fill the contents of pip freeze below -->
<details>
<summary><b><code>pip freeze</code> Output:</b></summary>
<p>

```
amqp==5.0.1
bidalgo-common==0.0
billiard==3.6.3.0
boto3==1.10.45
botocore==1.13.45
celery==5.0.0
certifi==2020.6.20
chardet==3.0.4
click==7.1.2
click-didyoumean==0.0.3
click-repl==0.1.6
colormath==3.0.0
common==0.1
coverage==5.0.3
creative-analysis===-module.-creative-analysis.version-.from.-tmp-pip-req-build-v4szy4xr-creative-analysis-version.py-
custom-stats==0.1
cycler==0.10.0
dataclasses==0.6
dataclasses-json==0.3.7
decorator==4.4.2
docutils==0.15.2
idna==2.8
importlib-metadata==2.0.0
jmespath==0.10.0
joblib==0.16.0
kiwisolver==1.2.0
kombu==5.0.2
marshmallow==3.8.0
marshmallow-enum==1.5.1
matplotlib==3.1.1
mypy-extensions==0.4.3
mysqlclient==2.0.1
networkx==2.5
nlp-tools==0.0
nltk==3.4.5
numpy==1.16.5
pandas==0.25.1
patsy==0.5.1
Pillow==5.4.1
prompt-toolkit==3.0.7
pydantic==1.6.1
PyMySQL==0.9.3
pyparsing==2.4.7
python-dateutil==2.8.1
python-dotenv==0.10.3
pytz==2020.1
redis==3.3.11
requests==2.22.0
s3transfer==0.2.1
scikit-learn==0.21.3
scipy==1.3.1
six==1.15.0
SQLAlchemy==1.3.19
statsmodels==0.11.0
stringcase==1.2.0
text2digits==0.0.9
typing-extensions==3.7.4.3
typing-inspect==0.6.0
urllib3==1.25.10
vine==5.0.0
wcwidth==0.2.5
Werkzeug==1.0.1
zipp==3.2.0

```

</p>
</details>

### Other Dependencies
<!--
Please provide system dependencies, configuration files
and other dependency information if applicable
-->
<details>
<p>
Status of node rabbit@f1b3c114b8d6 ...
Runtime

OS PID: 275
OS: Linux
Uptime (seconds): 759
Is under maintenance?: false
RabbitMQ version: 3.8.9
Node name: rabbit@f1b3c114b8d6
Erlang configuration: Erlang/OTP 23 [erts-11.1] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:64]
Erlang processes: 669 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

</p>
</details>

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.
-->

<details>
<p>

task.py:
```
import os
from celery import Celery
from time import sleep

celery_app = Celery("worker", broker=os.environ['CELERY_BROKER_URL'])

@celery_app.task()
def run():
    sleep(60)
    return

```
run.py:
```
from task import run

if __name__ == '__main__':
    run.apply_async()
    run.apply_async()
```

1. IMPORTANT - Start broker - **bug happens only when first tasks run after broker starts.**

2. Run celery:
```$ celery -A task worker --pool=prefork --concurrency=2```

3. Run tasks:
```$ python run.py```

4. inspect active tasks:
```$ celery -A task inspect active```

5. inspect reserved tasks:
```$ celery -A task inspect reserved```


</p>
</details>

# Expected Behavior
<!-- Describe in detail what you expect to happen -->

Worker is running 2 active tasks

# Actual Behavior
<!--
Describe in detail what actually happened.
Please include a backtrace and surround it with triple backticks (```).
In addition, include the Celery daemon logs, the broker logs,
the result backend logs and system logs below if they will help us debug
the issue.
-->

The worker is running only 1 active task and you can see 1 reserved task.

Same test performed on 4.4.7 (cliffs) worked as expected

Even in 5.0.0 when you run again

3. Run tasks:
```$ python run.py```

4. inspect active tasks:
```$ celery -A task inspect active```

5. inspect reserved tasks:
```$ celery -A task inspect reserved```

It will work as expected - you will see 2 active tasks and 0 reserved


Discussion:
Seeing the same on Ubuntu 20 and Python 3.7. 

Started testing v5 today and found my performance tanked. 
Facing the same issue on ubuntu 16.04 and python3.6. 
Celery v5.0.0 executes a single task while others sit idle as reserved. Shifting back to 4.4.7 gives the expected behavior.
I'll try to reproduce the problem.

> Started testing v5 today and found my performance tanked.

In the future, if you can, please try our release candidates.
I'd rather we find these issues before GA. :smile: 
Seeing the same here in the Python 3.6.10 Docker image
Seeing the same issue with oraculum ( https://pagure.io/fedora-qa/oraculum ).

Celery 4.4.7 works as expected. Only one worker is getting tasks done with Celery 5.0.0 . We're using redis backend for celery.
Same here. When is this fix expected to be released?

Nvm this is just an issue, not an MR.
Same here, returned to 4.4.7 and the work can run tasks concurrently. Tested in CentOS 7 - Python 3.6.8 and macOS 10.14 - Python 3.7.1.

My celery code:

```
import time
from celery import Celery


app = Celery('hello', broker='redis://localhost:6379/3')

@app.task
def long(x):
    print(f'sleep {x}')
    time.sleep(x)
    print('done sleep')
```

With 5.0.1:

![image](https://user-images.githubusercontent.com/405972/97071320-2c192580-1612-11eb-8b8a-6e21965c3201.png)

With 4.4.7:

![image](https://user-images.githubusercontent.com/405972/97071332-363b2400-1612-11eb-9287-80499b80db58.png)

Took me two days to realize this is not a problem about configuration, system, and my knowledge about celery. 🤦‍♂️
I also experienced this problem. Upgraded from 4.4.7 to 5.0.1 and performance plummeted because workers aren't executing tasks concurrently. Reverted back to 4.4.7 and the problem went away immediately.
That's kinda of a breaking issue. Any chance that this could be addressed as a separate small bugfix release? 5.0.2 with just this fix?
I've been digging into this today and have some observations:
 * `inspect active` shows that only a single worker PID is running tasks
 * sometimes two tasks will run concurrently with different worker PIDs if the task queue was empty
 * after tasks begin to serialize, only one worker will handle tasks
 * On my box with concurrency >= 6, worker 6 appears to always win whatever race to whatever mutex this is
   * this is stable across restarts of the worker, I've not checked across reboots
   * the other worker which sometimes gets tasks also seems to be deterministic - 2 for concurrency 6,7; 7 for concurrency > 7
 * this misbehaviour is relevant to prefork only AFAICT - gevent and eventlet seem fine

I'm running a tweaked variant of @reorx's MRTC so I can do `celery -A app worker` and have it queue tasks for me at the same time as starting a worker, or running `python app.py` to queue more tasks.

```python
import sys
import time
from celery import Celery

app = Celery("app", broker="redis://", backend="redis://")

@app.task
def long_(x):
    print(f"sleep {x}")
    time.sleep(x)
    print("done sleep")

if "inspect" not in sys.argv:
    for _ in range(10):
        long_.s(1).delay()
```
I am able to trick the prefork worker into running tasks concurrently if I shuffle the list of events returned by the kombu poller to the hub, prior to the event handling loop around
https://github.com/celery/kombu/blob/b01a448a4c85753600cd331bd8af618de4b0ee67/kombu/asynchronous/hub.py#L305

A bit of instrumentation around there suggest that the `poll_timeout` is being set to a pretty high value and we're seemingly always ending up choosing to deliver any pending request to the same one or two workers which happen to sort at the start of the list of events. This seems like some sort of failure to round robin in either the hub or celery's use of it, but I can't spot what could have caused it between 4.4.7 and now. Perhaps @thedrow or @matusvalo have some insight?

Resolving patch:
diff --git a/celery/concurrency/asynpool.py b/celery/concurrency/asynpool.py
--- a/celery/concurrency/asynpool.py
+++ b/celery/concurrency/asynpool.py
@@ -84,6 +84,7 @@ def unpack_from(fmt, iobuf, unpack=unpack):  # noqa
 
 SCHED_STRATEGIES = {
     None: SCHED_STRATEGY_FAIR,
+    'default': SCHED_STRATEGY_FAIR,
     'fast': SCHED_STRATEGY_FCFS,
     'fcfs': SCHED_STRATEGY_FCFS,
     'fair': SCHED_STRATEGY_FAIR,


Test patch: