Issue 1:
Checklist

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

Report

software -> celery:4.0.2 (latentcall) kombu:4.0.2 py:3.5.0
billiard:3.5.0.2 py-amqp:2.1.4
platform -> system:Darwin arch:64bit imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:rpc

imports:
('project.myapp.tasks',)
task_annotations: {
'tasks.add': {'rate_limit': '10/s'}}
broker_url: 'amqp://guest:********@localhost:5672//'
result_backend: 'rpc'

Steps to reproduce

Start default rabbitmq-server and a default worker

Have two tasks, one called inside the other:

@celery_app.task
def super_task():
async_res = sub_task.apply_async()
while not async_res.ready():
time.sleep(1)

if async_res.failed():
raise async_res.result
else:
return async_res.result

@celery_app.task
def sub_task():
raise Exception('Controlled error')

Call for super_task

r = super_task.apply_async()

try:
r.get()
except Exception as exc:
if exc.__class__.__module__ != 'builtins':
assert 0, 'Exception type is not builtin'

optionally: download the attached project, install celery and run the file test_raising.py

celery_project.zip

Expected behavior

The subtask exception of type Exception from python builtins module to be arisen up to the main process and the assert check successful

Actual behavior

The arisen exception is not builtin is of class celery.backends.base.Exception instead, and if you check the worker log a super_task task handler raised an error:

<MaybeEncodingError: Error sending result: '"(1, <ExceptionInfo: Exception('Controlled error',)>, None)"'. Reason: ''PicklingError("Can\'t pickle <class \'celery.backends.base.Exception\'>: attribute lookup Exception on celery.backends.base failed",)''.>

Issue 2:
Checklist

I have verified that the issue exists against the master branch of Celery.

[] This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version: 4.4.0

celery report Output:

keep having celery: error: unrecognized arguments: report

but the error is in when calling to celery and not from celery it self

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.6.1

Minimal Celery Version: 4.4.0

Minimal Kombu Version: 4.6.7

Minimal Broker Version: 3.8.3

Minimal Result Backend Version: 5.0.3

Minimal OS and/or Kernel Version: Mac 10.14.6

Minimal Broker Client Version: 3.3.11

Minimal Result Backend Client Version: 3.3.11

Python Packages

pip freeze Output:

kombu==4.6.7
amqp==2.5.2
celery==4.4.0
redis==3.3.11

Other Dependencies

N/A

Minimally Reproducible Test Case

Its not a test case but please run the python file here

https://gist.github.com/ericman93/2ec31270ea68b3cfd56338039c766e9c

Expected Behavior

Taks to run

Actual Behavior

Trying to run this pipeline

eric.celery.core.print_message('START!') | %(dummy([print_message('Starting subtask A'), eric.celery.tasks.change_project_status_task('ad56c520f9a74173ad3e6446f19712b8', 'Subtask A')]) | %(dummy([print_message('Starting Subtask B'), eric.tasks.change_project_status_task('ad56c520f9a74173ad3e6446f19712b8', 'Subtask B'), eric.tasks.subtask_b('ad56c520f9a74173ad3e6446f19712b8', '458db63427bd48e98f9b3d27204589a9', 2, '946f464babca43de8bc03e5c1202f813')]) | %(dummy([print_message('Starting Subtask C'), eric.tasks.change_project_status_task('ad56c520f9a74173ad3e6446f19712b8', 'Subtask C'), eric.tasks.subtask_c('ad56c520f9a74173ad3e6446f19712b8', 'C')]) | print_message('Starting next session') | %(dummy([%(dummy([print_message('Starting subtask D-I'), eric.tasks.change_project_status_task('ad56c520f9a74173ad3e6446f19712b8', 'Subtask D'), eric.tasks.subtask_d('974fb9cb12e04a3a856c4438b5bb33b7')]) | %dummy([print_message('Starting Subtask D-I'), eric.tasks.change_project_status_task('ad56c520f9a74173ad3e6446f19712b8', 'Subtask E'), eric.tasks.subtask_e('aa107a89e6d84aa1999269725f5a6b11', 2), eric.tasks.subtask_e('ce2d6f8dc6494222979b00582f75a0eb', 2), eric.tasks.subtask_e('35d2c5f0d04a4c7cb48751ddec3e469d', 2), eric.tasks.subtask_e('410318c5579046cdbf009be243e4596c', 2)])), %(dummy([print_message('Starting subtask D-II'), eric.tasks.change_project_status_task('ad56c520f9a74173ad3e6446f19712b8', 'Subtask D'), eric.tasks.subtask_d('0cd1a4cd01774644aa9ee8cc1fbac834')]) | %dummy([print_message('Starting Subtask E-II'), eric.tasks.change_project_status_task('ad56c520f9a74173ad3e6446f19712b8', 'Subtask E'), eric.tasks.subtask_e('c6d0947ac9214d68a1b3ef889a2fe2b2', 2), eric.tasks.subtask_e('81a7e39833e54fdcba9dbe0590354954', 2), eric.tasks.subtask_e('38f80253760c4f8d8747e667dfc90914', 2), eric.tasks.subtask_e('5249d50e4c0342e8abe3d343f9998361', 2)]))]) | %(dummy([print_message('Staring subtask F'), eric.tasks.change_project_status_task('ad56c520f9a74173ad3e6446f19712b8', 'Subtask F'), eric.tasks.subtask_f('ad56c520f9a74173ad3e6446f19712b8')]) | %(dummy([print_message('Starting subtask G'), eric.tasks.change_project_status_task('ad56c520f9a74173ad3e6446f19712b8', 'Subtask G'), eric.tasks.subtask_g_1('ad56c520f9a74173ad3e6446f19712b8'), eric.tasks.subtask_g_2('ad56c520f9a74173ad3e6446f19712b8')]) | %dummy([print_message('Final sub task'), eric.tasks.change_project_status_task('ad56c520f9a74173ad3e6446f19712b8', 'Complete'), eric.tasks.finish_1('ad56c520f9a74173ad3e6446f19712b8'), eric.tasks.finish_2('ad56c520f9a74173ad3e6446f19712b8')])))))))

When using Rabbit as a broker it tasks 20 seconds to run applay_async and then the tasks are not starting in celery.

When using Redis as a broker this exception raising

Traceback (most recent call last):
File "/Users/eric/.local/share/virtualenvs/romee-vHL_yubm/lib/python3.6/site-packages/redis/connection.py", line 666, in send_packed_command
sendall(self._sock, item)
File "/Users/eric/.local/share/virtualenvs/romee-vHL_yubm/lib/python3.6/site-packages/redis/_compat.py", line 8, in sendall
return sock.sendall(*args, **kwargs)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/eric/.local/share/virtualenvs/romee-vHL_yubm/lib/python3.6/site-packages/kombu/connection.py", line 439, in _reraise_as_library_errors
yield
File "/Users/eric/.local/share/virtualenvs/romee-vHL_yubm/lib/python3.6/site-packages/kombu/connection.py", line 518, in _ensured
return fun(*args, **kwargs)
File "/Users/eric/.local/share/virtualenvs/romee-vHL_yubm/lib/python3.6/site-packages/kombu/messaging.py", line 203, in _publish
mandatory=mandatory, immediate=immediate,
File "/Users/eric/.local/share/virtualenvs/romee-vHL_yubm/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 608, in basic_publish
return self._put(routing_key, message, **kwargs)
File "/Users/eric/.local/share/virtualenvs/romee-vHL_yubm/lib/python3.6/site-packages/kombu/transport/redis.py", line 789, in _put
client.lpush(self._q_for_pri(queue, pri), dumps(message))
File "/Users/eric/.local/share/virtualenvs/romee-vHL_yubm/lib/python3.6/site-packages/redis/client.py", line 1679, in lpush
return self.execute_command('LPUSH', name, *values)
File "/Users/eric/.local/share/virtualenvs/romee-vHL_yubm/lib/python3.6/site-packages/redis/client.py", line 838, in execute_command
conn.send_command(*args)
File "/Users/eric/.local/share/virtualenvs/romee-vHL_yubm/lib/python3.6/site-packages/redis/connection.py", line 687, in send_command
check_health=kwargs.get('check_health', True))
File "/Users/eric/.local/share/virtualenvs/romee-vHL_yubm/lib/python3.6/site-packages/redis/connection.py", line 679, in send_packed_command
(errno, errmsg))
redis.exceptions.ConnectionError: Error 32 while writing to socket. Broken pipe.

This is really gets stuck at producer.publish in amqp.py

If I make the pipeline smaller, everthing works fine