Issue 1:
Some notes about configuration:

pip install celery -U (3.0.19 and deps, with python-dateutil 1.5.0)

CELERYD_TASK_TIME_LIMIT = 1800 # 30 minutes

CELERYD_MAX_TASKS_PER_CHILD = 100000

Here's some (hopefully) helpful diagnostic information:

(Pdb) hp.heap().get_rp()
Reference Pattern by <[dict of] class>.
0: _ --- [-] 425547 (__builtin__.cell | __builtin__.weakcallableproxy | __bu...
1: a [-] 34934 dict (no owner): 0x38e1ee0*32, 0x4fd8040*3, 0x4fd8160*16...
2: aa ---- [-] 15515 dict of billiard.pool.ApplyResult: 0x3aabd10..., 0x4fae190
3: a3 [-] 15515 billiard.pool.ApplyResult: 0x37fc0d0, 0x3a4c410...
4: a4 ------ [-] 15490 tuple: 0x1240f50*1, 0x38b7850*1, 0x3a4c190*1...
5: a5 [-] 15490 celery.utils.timer2.Entry: 0x38aed50, 0x38aedb8...
6: a6 -------- [-] 15490 tuple: 0x38bc280*3, 0x38bc320*3, 0x3a540a0*3...
7: a7 [+] 1 list: 0x38920e0*15491
8: a5b ------- [-] 7 dict of billiard.pool.ApplyResult: 0xcfccd90..., 0xcfd3cd0
9: a5ba [+] 7 billiard.pool.ApplyResult: 0xcfccd90, 0xcfd3690...
<Type e.g. '_.more' for more.>
(Pdb) hp.heap().get_rp()
Reference Pattern by <[dict of] class>.
0: _ --- [-] 796231 (__builtin__.cell | __builtin__.weakcallableproxy | __bu...
1: a [-] 66062 dict (no owner): 0x38e1ee0*40, 0x4fd8040*3, 0x4fd8160*16...
2: aa ---- [-] 21741 dict of billiard.pool.ApplyResult: 0x3aabd10..., 0x4fae190
3: a3 [-] 21741 billiard.pool.ApplyResult: 0x37fc0d0, 0x3a4c410...
4: a4 ------ [-] 21709 tuple: 0x1240f50*1, 0x38b7850*1, 0x3a4c190*1...
5: a5 [-] 21709 celery.utils.timer2.Entry: 0x38aed50, 0x38aedb8...
6: a6 -------- [-] 21709 tuple: 0x38bc280*3, 0x38bc320*3, 0x3a540a0*3...
7: a7 [+] 1 list: 0x38920e0*21709
8: a5b ------- [-] 8 dict of billiard.pool.ApplyResult: 0x10789a50...
9: a5ba [+] 8 billiard.pool.ApplyResult: 0x10789a50, 0x10789b90...
<Type e.g. '_.more' for more.>
(Pdb) hp.heap().get_rp()
Reference Pattern by <[dict of] class>.
0: _ --- [-] 2767466 (__builtin__.cell | __builtin__.weakcallableproxy | __b...
1: a [-] 213471 dict (no owner): 0x38e1ee0*10, 0x4fd8040*3...
2: aa ---- [-] 51209 dict of billiard.pool.ApplyResult: 0x3aabd10..., 0x4fae190
3: a3 [-] 51209 billiard.pool.ApplyResult: 0x37fc0d0, 0x3a4c410...
4: a4 ------ [-] 51207 tuple: 0x1240f50*1, 0x38b7850*1, 0x3a4c190*1...
5: a5 [-] 51207 celery.utils.timer2.Entry: 0x38aed50, 0x38aedb8...
6: a6 -------- [-] 51207 tuple: 0x38bc280*3, 0x38bc320*3, 0x3a540a0*3...
7: a7 [+] 1 list: 0x38920e0*51212
8: a5b ------- [-] 8 dict of billiard.pool.ApplyResult: 0x21c8bcd0...
9: a5ba [+] 8 billiard.pool.ApplyResult: 0x21c8bcd0, 0x21d46150...
<Type e.g. '_.more' for more.>
(Pdb) heap = hp.heap()
(Pdb) heap[0]
Partition of a set of 367501 objects. Total size = 159517240 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 367501 100 159517240 100 159517240 100 dict (no owner)
(Pdb) heap[0].byrcs
Partition of a set of 367501 objects. Total size = 159517240 bytes.
Index Count % Size % Cumulative % Referrers by Kind (class / dict of class)
0 45910 12 48113680 30 48113680 30 celery.worker.job.Request
1 91820 25 47047712 29 95161392 60 celery.worker.job.Request, dict (no owner)
2 137730 37 38564400 24 133725792 84 kombu.transport.librabbitmq.Message
3 45942 13 12863760 8 146589552 92 celery.utils.timer2.Entry
4 45911 12 12855080 8 159444632 100 dict (no owner)
5 138 0 39408 0 159484040 100 dict of guppy.etc.Glue.Owner
6 42 0 24048 0 159508088 100 dict of guppy.etc.Glue.Share
7 3 0 6984 0 159515072 100 dict of guppy.heapy.UniSet.Summary_str
8 1 0 1048 0 159516120 100 __builtin__.cell
9 2 0 560 0 159516680 100 <Nothing>
<2 more rows. Type e.g. '_.more' to view.>

Issue 2:
the dependency on future-0.18.2 adds many packages that squat on py3 stdlib names eg:

/home/graingert/.virtualenvs/celery2/lib/python2.7/site-packages/_dummy_thread/*
/home/graingert/.virtualenvs/celery2/lib/python2.7/site-packages/_markupbase/*
/home/graingert/.virtualenvs/celery2/lib/python2.7/site-packages/_thread/*
/home/graingert/.virtualenvs/celery2/lib/python2.7/site-packages/builtins/*
/home/graingert/.virtualenvs/celery2/lib/python2.7/site-packages/copyreg/*
/home/graingert/.virtualenvs/celery2/lib/python2.7/site-packages/future-0.18.2.dist-info/*
/home/graingert/.virtualenvs/celery2/lib/python2.7/site-packages/future/*
/home/graingert/.virtualenvs/celery2/lib/python2.7/site-packages/html/*
/home/graingert/.virtualenvs/celery2/lib/python2.7/site-packages/http/*
/home/graingert/.virtualenvs/celery2/lib/python2.7/site-packages/libfuturize/*
/home/graingert/.virtualenvs/celery2/lib/python2.7/site-packages/libpasteurize/*
/home/graingert/.virtualenvs/celery2/lib/python2.7/site-packages/past/*
/home/graingert/.virtualenvs/celery2/lib/python2.7/site-packages/queue/*
/home/graingert/.virtualenvs/celery2/lib/python2.7/site-packages/reprlib/*
/home/graingert/.virtualenvs/celery2/lib/python2.7/site-packages/socketserver/*
/home/graingert/.virtualenvs/celery2/lib/python2.7/site-packages/tkinter/*
/home/graingert/.virtualenvs/celery2/lib/python2.7/site-packages/winreg/*
/home/graingert/.virtualenvs/celery2/lib/python2.7/site-packages/xmlrpc/*

this breaks packages that rely on feature detection of these modules.

I propose replacing future with six because:

future does not provide a wheel, six has a wheel

future last release Oct 31, 2019, six released Oct 31, 2019

future is 829.2 kB, six is (11.0 kB)

celery already depends on six, via kombu and importlib-metadata, and so swapping out future with six would save 829.2 kB, rather than just 818.2 kB

Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version:

celery report Output:

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown

Minimal Celery Version: N/A or Unknown

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

Other Dependencies

N/A

Minimally Reproducible Test Case

Expected Behavior

Actual Behavior