Issue 1:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

[y] I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

[] I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

#6791

#1997

Possible Duplicates

None

Environment & Settings

Celery version:

celery report Output:

software -> celery:5.1.2 (sun-harmonics) kombu:5.1.0 py:3.8.5
billiard:3.6.4.0 redis:3.5.3
platform -> system:Linux arch:64bit, ELF
kernel version:5.8.0-59-generic imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:disabled

broker_url: 'redis://localhost:6379/0'
deprecated_settings: None

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown

Minimal Celery Version: N/A or Unknown

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: redis in docker

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: Ubuntu 20.04

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

amqp==5.0.6
billiard==3.6.4.0
celery @ file:///tmp/x/celery
click==7.1.2
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.2.0
kombu==5.1.0
prompt-toolkit==3.0.19
pytz==2021.1
redis==3.5.3
six==1.16.0
vine==5.0.0
wcwidth==0.2.5

Other Dependencies

N/A

Minimally Reproducible Test Case

from celery import Celery

app = Celery('tasks', broker='redis://localhost:6379/0')

@app.task
def f():
print('hello hello world')

if __name__ == '__main__':
f.apply_async()

Expected Behavior

only one line of output

-------------- [queues]
.> celery exchange=celery(direct) key=celery

[2021-06-30 19:28:02,606: WARNING/ForkPoolWorker-4] hello hello world

Actual Behavior

prints an empty line

-------------- [queues]
.> celery exchange=celery(direct) key=celery

[2021-06-30 19:28:02,606: WARNING/ForkPoolWorker-4] hello hello world
[2021-06-30 19:28:02,607: WARNING/ForkPoolWorker-4]

when printing from the task function, each output causes a newline to be printed to stderr. This was introduced in version 5.1.* in #6791 by removing this line: 51f5b01#diff-116905b4c2ccc7e0172e26700ba1a0a0270b2fe1f830c47244dc1df07c805883L226

I think there are different options how to address this:

just add the line back as it was -- but this would go against the initaly intention in #6791 of removing it to don't do any string manipulation in this function

use something like re.sub('\n|\r', '', data ) instead, to keep the whitespace or an endswith check or a more complex regex?

address this somewhere else, so no newline characters are passed to .write() (I would need a hint where to do this though)

I am happy to send a PR to fix this if needed.

Issue 2:
Checklist

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

Subsequent groups within a chain fail #3585

Nested group(chain(group)) fails #5496

group([chain(group(task(), task()), group(...)), chain(group(...), group(...))]) construct never finishes #2354

Canvas with group-task-task-group does not work #5354

Possible Duplicates

Nested group-chain-group structure never succeeds #2573

Environment & Settings

Celery version:

celery report Output:

software -> celery:4.3.0 (rhubarb) kombu:4.5.0 py:3.6.5
billiard:3.6.0.0 redis:3.2.1
platform -> system:Linux arch:64bit
kernel version:4.9.0-9-amd64 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis:///

broker_url: 'redis://localhost:6379//'
result_backend: 'redis:///'

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.6

Minimal Celery Version: 4.3

Minimal Kombu Version: 4.5.0

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: Redis

Minimal OS and/or Kernel Version: Debian Stretch

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

amqp==2.4.2
billiard==3.6.0.0
celery==4.3.0
celery-eternal==0.1.1
celery-singleton==0.1.3
certifi==2018.4.16
kombu==4.5.0
pytz==2019.1
redis==3.2.1
vine==1.3.0

Other Dependencies

N/A

Minimally Reproducible Test Case

from time import sleep

from celery import Celery
from celery import group

app = Celery('tasks', broker = 'redis://')
app.conf['result_backend'] = app.conf.broker_url

@app.task
def prod(x, y):
return x*y

@app.task
def subtract(args):
return args[0]-args[1]

@app.task(shared=False)
def identity(args):
"""Identity task (returns its input)."""
return args

x = (
group( prod.s(1, 1), prod.s(2, 1) )
|
identity.s()
|
subtract.s()
|
group( prod.s(5), prod.s(6) )
)
r = x.delay()
sleep(10)
print(r.waiting())

# Another case
x = (
group( prod.s(1, 1), prod.s(2, 1) )
|
subtract.si((3,4))
|
subtract.si((4,3))
|
subtract.si((6,5))
|
group( prod.si(5, 6), prod.si(6, 5) )
)
r = x.delay()
sleep(10)
print(r.waiting())

# However this works
# x = (
# group( prod.s(1, 1), prod.s(2, 1) )
# |
# subtract.si((3,4))
# |
# group( prod.si(5, 6), prod.si(6, 5) )
# )
# r = x.delay()
# print([p.result for p in r.results])
# prints [30, 30]

# The adding more than one component in between 2 groups
# in a chain is causing the canvas to wait right before the final group

Expected Behavior

Above code snippet should return a list of 2 values

([5, 6] in for first case, [30, 30] for second case)

Actual Behavior

The code never never returns, keeps waiting

Celery worker logs for the first case mentioned

[2019-05-10 21:12:42,361: INFO/MainProcess] Connected to redis://localhost:6379//
[2019-05-10 21:12:42,366: INFO/MainProcess] mingle: searching for neighbors
[2019-05-10 21:12:43,380: INFO/MainProcess] mingle: all alone
[2019-05-10 21:12:43,393: INFO/MainProcess] celery@debian ready.
[2019-05-10 21:13:43,557: INFO/MainProcess] Received task: celery_task.prod[e11748a0-dca2-41fb-b799-a2ae7879c9fd]
[2019-05-10 21:13:43,559: INFO/MainProcess] Received task: celery_task.prod[6ec1c88f-cca7-417c-8ff1-eacd9bdef7a1]
[2019-05-10 21:13:43,566: INFO/ForkPoolWorker-2] Task celery_task.prod[6ec1c88f-cca7-417c-8ff1-eacd9bdef7a1] succeeded in 0.0064055299990286585s: 2
[2019-05-10 21:13:43,585: INFO/ForkPoolWorker-1] Task celery_task.prod[e11748a0-dca2-41fb-b799-a2ae7879c9fd] succeeded in 0.02549504399939906s: 1
[2019-05-10 21:13:43,586: INFO/MainProcess] Received task: celery_task.identity[87148533-8da1-4fc6-bbfe-608470cb3c1a]
[2019-05-10 21:13:43,588: INFO/ForkPoolWorker-1] Task celery_task.identity[87148533-8da1-4fc6-bbfe-608470cb3c1a] succeeded in 0.0011925399994652253s: [2, 1]
[2019-05-10 21:13:43,588: INFO/MainProcess] Received task: celery_task.subtract[e907af43-a886-45ea-bfe3-f709c8267e5d]
[2019-05-10 21:13:43,589: INFO/ForkPoolWorker-1] Task celery_task.subtract[e907af43-a886-45ea-bfe3-f709c8267e5d] succeeded in 0.00028916799965372775s: 1

Considering the first case mentioned in the code snippet, you can see that value of the task before the second group, subtract, is returned but the last group (consisting 2 prods) is never received, even after waiting many minutes. (I put the sleep(10) just as a proxy for some wait)