Repository: celery/celery

Issue:
Continuous memory leak
There is a memory leak in the parent process of Celery's worker.
It is not a child process executing a task.
It happens suddenly every few days.
Unless you stop Celery, it consumes server memory in tens of hours.

This problem happens at least in Celery 4.1, and it also occurs in Celery 4.2.
Celery is running on Ubuntu 16 and brokers use RabbitMQ.

![memory](https://user-images.githubusercontent.com/280983/41805959-fcdd7700-76ed-11e8-9e02-db322818e924.jpeg)




Discussion:
Are you using Canvas workflows? Maybe #4839 is related.

Also I assume you are using prefork pool for worker concurrency?
Thanks georgepsarakis.

I am not using workflow.
I use prefork concurrency 1 on single server.
The increase rate seems quite linear, quite weird. Is the worker processing tasks during this time period? Also, can you add a note with the complete command you are using to start the worker?
Yes. The worker continues to process the task normally.

The worker is started with the following command.

`/xxxxxxxx/bin/celery worker --app=xxxxxxxx --loglevel=INFO --pidfile=/var/run/xxxxxxxx.pid`
This problem is occurring in both the production environment and the test environment.
I can add memory profile and test output to the test environment.
If there is anything I can do, please say something.
We need to understand what the worker is running during the time that the memory increase is observed. Any information and details you can possibly provide would definitely. It is also good that you can reproduce this.
Although it was a case occurred at a timing different from the graph, the next log was output at the timing when the memory leak started.

```
[2018-02-24 07:50:52,953: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
File "/xxxxxxxx/lib/python3.5/site-packages/celery/worker/consumer/consumer.py", line 320, in start
blueprint.start(self)
File "/xxxxxxxx/lib/python3.5/site-packages/celery/bootsteps.py", line 119, in start
step.start(parent)
File "/xxxxxxxx/lib/python3.5/site-packages/celery/worker/consumer/consumer.py", line 596, in start
c.loop(*c.loop_args())
File "/xxxxxxxx/lib/python3.5/site-packages/celery/worker/loops.py", line 88, in asynloop
next(loop)
File "/xxxxxxxx/lib/python3.5/site-packages/kombu/async/hub.py", line 293, in create_loop
poll_timeout = fire_timers(propagate=propagate) if scheduled else 1
File "/xxxxxxxx/lib/python3.5/site-packages/kombu/async/hub.py", line 136, in fire_timers
entry()
File "/xxxxxxxx/lib/python3.5/site-packages/kombu/async/timer.py", line 68, in __call__
return self.fun(*self.args, **self.kwargs)
File "/xxxxxxxx/lib/python3.5/site-packages/kombu/async/timer.py", line 127, in _reschedules
return fun(*args, **kwargs)
File "/xxxxxxxx/lib/python3.5/site-packages/kombu/connection.py", line 290, in heartbeat_check
return self.transport.heartbeat_check(self.connection, rate=rate)
File "/xxxxxxxx/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 149, in heartbeat_check
return connection.heartbeat_tick(rate=rate)
File "/xxxxxxxx/lib/python3.5/site-packages/amqp/connection.py", line 696, in heartbeat_tick
self.send_heartbeat()
File "/xxxxxxxx/lib/python3.5/site-packages/amqp/connection.py", line 647, in send_heartbeat
self.frame_writer(8, 0, None, None, None)
File "/xxxxxxxx/lib/python3.5/site-packages/amqp/method_framing.py", line 166, in write_frame
write(view[:offset])
File "/xxxxxxxx/lib/python3.5/site-packages/amqp/transport.py", line 258, in write
self._write(s)
ConnectionResetError: [Errno 104] Connection reset by peer
[2018-02-24 08:49:12,016: INFO/MainProcess] Connected to amqp://xxxxxxxx:**@xxx.xxx.xxx.xxx:5672/xxxxxxxx
```

It seems that it occurred when the connection with RabbitMQ was temporarily cut off.
@marvelph so it occurs during RabbitMQ reconnections? Perhaps these issues are related: 
- https://github.com/celery/kombu/issues/843
- https://github.com/celery/celery/pull/4839#issuecomment-399633253
Yes.
It seems that reconnection triggers it.
It looks like I'm having the same issue... It is so hard for me to find out what triggers it and why there is a memeory leak. It annoys me for at least a month. I fallback to used celery 3 and everything is fine.

For the memory leak issue, I'm using ubuntu 16, celery 4.1.0 with rabbitmq. I deployed it via docker.

The memory leak is with MainProcess not ForkPoolWorker. The memory usage of ForkPoolWorker is normal, but memory usage of MainProcess is always increasing. For five seconds, around 0.1MB memeory is leaked. The memory leak doesn't start after the work starts immediatly but maybe after one or two days.

I used gdb and pyrasite to inject the running process and try to ```gc.collect()```, but nothing is collected.

I checked the log, the ```consumer: Connection to broker lost. Trying to re-establish the connection...``` did happens, but for now I'm not sure this is the time when memory leak happens.

Any hints for debugging this issue and to find out what really happens? Thanks.
Since @marvelph mentioned it may relate with rabbitmq reconnection, I try to stop my rabbitmq server. The  memory usage did increase after each reconnection, following is the log. So I can confirm this https://github.com/celery/kombu/issues/843 issue.

But after the connection is reconnected, the memory usage stops to gradually increase. So I'm not sure this is the reason for memory leak.

I will try to use redis to figure out whether this memory leak issue relates wtih rabbitmq or not.

```
[2018-06-25 02:43:33,456: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/app/.heroku/python/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 316, in start
    blueprint.start(self)
  File "/app/.heroku/python/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/app/.heroku/python/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 592, in start
    c.loop(*c.loop_args())
  File "/app/.heroku/python/lib/python3.6/site-packages/celery/worker/loops.py", line 91, in asynloop
    next(loop)
  File "/app/.heroku/python/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 354, in create_loop
    cb(*cbargs)
  File "/app/.heroku/python/lib/python3.6/site-packages/kombu/transport/base.py", line 236, in on_readable
    reader(loop)
  File "/app/.heroku/python/lib/python3.6/site-packages/kombu/transport/base.py", line 218, in _read
    drain_events(timeout=0)
  File "/app/.heroku/python/lib/python3.6/site-packages/amqp/connection.py", line 491, in drain_events
    while not self.blocking_read(timeout):
  File "/app/.heroku/python/lib/python3.6/site-packages/amqp/connection.py", line 496, in blocking_read
    frame = self.transport.read_frame()
  File "/app/.heroku/python/lib/python3.6/site-packages/amqp/transport.py", line 243, in read_frame
    frame_header = read(7, True)
  File "/app/.heroku/python/lib/python3.6/site-packages/amqp/transport.py", line 418, in _read
    s = recv(n - len(rbuf))
ConnectionResetError: [Errno 104] Connection reset by peer
[2018-06-25 02:43:33,497: ERROR/MainProcess] consumer: Cannot connect to amqp://***:**@***:***/***: [Errno 111] Connection refused.
Trying again in 2.00 seconds...

[2018-06-25 02:43:35,526: ERROR/MainProcess] consumer: Cannot connect to amqp://***:**@***:***/***: [Errno 111] Connection refused.
Trying again in 4.00 seconds...

[2018-06-25 02:43:39,560: ERROR/MainProcess] consumer: Cannot connect to amqp://***:**@***:***/***: [Errno 111] Connection refused.
Trying again in 6.00 seconds...

[2018-06-25 02:43:45,599: ERROR/MainProcess] consumer: Cannot connect to amqp://***:**@***:***/***: [Errno 111] Connection refused.
Trying again in 8.00 seconds...

[2018-06-25 02:43:53,639: ERROR/MainProcess] consumer: Cannot connect to amqp://***:**@***:***/***: [Errno 111] Connection refused.
Trying again in 10.00 seconds...

[2018-06-25 02:44:03,680: ERROR/MainProcess] consumer: Cannot connect to amqp://***:**@***:***/***: [Errno 111] Connection refused.
Trying again in 12.00 seconds...

[2018-06-25 02:44:15,743: ERROR/MainProcess] consumer: Cannot connect to amqp://***:**@***:***/***: [Errno 111] Connection refused.
Trying again in 14.00 seconds...

[2018-06-25 02:44:29,790: ERROR/MainProcess] consumer: Cannot connect to amqp://***:**@***:***/***: [Errno 111] Connection refused.
Trying again in 16.00 seconds...

[2018-06-25 02:44:45,839: ERROR/MainProcess] consumer: Cannot connect to amqp://***:**@***:***/***: [Errno 111] Connection refused.
Trying again in 18.00 seconds...

[2018-06-25 02:45:03,890: ERROR/MainProcess] consumer: Cannot connect to amqp://***:**@***:***/***: [Errno 111] Connection refused.
Trying again in 20.00 seconds...

[2018-06-25 02:45:23,943: ERROR/MainProcess] consumer: Cannot connect to amqp://***:**@***:***/***: [Errno 111] Connection refused.
Trying again in 22.00 seconds...

[2018-06-25 02:45:46,002: ERROR/MainProcess] consumer: Cannot connect to amqp://***:**@***:***/***: [Errno 111] Connection refused.
Trying again in 24.00 seconds...

[2018-06-25 02:46:10,109: INFO/MainProcess] Connected to amqp://***:**@***:***/***
[2018-06-25 02:46:10,212: INFO/MainProcess] mingle: searching for neighbors
[2018-06-25 02:46:10,291: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/app/.heroku/python/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 316, in start
    blueprint.start(self)
  File "/app/.heroku/python/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/app/.heroku/python/lib/python3.6/site-packages/celery/worker/consumer/mingle.py", line 40, in start
    self.sync(c)
  File "/app/.heroku/python/lib/python3.6/site-packages/celery/worker/consumer/mingle.py", line 44, in sync
    replies = self.send_hello(c)
  File "/app/.heroku/python/lib/python3.6/site-packages/celery/worker/consumer/mingle.py", line 57, in send_hello
    replies = inspect.hello(c.hostname, our_revoked._data) or {}
  File "/app/.heroku/python/lib/python3.6/site-packages/celery/app/control.py", line 132, in hello
    return self._request('hello', from_node=from_node, revoked=revoked)
  File "/app/.heroku/python/lib/python3.6/site-packages/celery/app/control.py", line 84, in _request
    timeout=self.timeout, reply=True,
  File "/app/.heroku/python/lib/python3.6/site-packages/celery/app/control.py", line 439, in broadcast
    limit, callback, channel=channel,
  File "/app/.heroku/python/lib/python3.6/site-packages/kombu/pidbox.py", line 315, in _broadcast
    serializer=serializer)
  File "/app/.heroku/python/lib/python3.6/site-packages/kombu/pidbox.py", line 290, in _publish
    serializer=serializer,
  File "/app/.heroku/python/lib/python3.6/site-packages/kombu/messaging.py", line 181, in publish
    exchange_name, declare,
  File "/app/.heroku/python/lib/python3.6/site-packages/kombu/messaging.py", line 203, in _publish
    mandatory=mandatory, immediate=immediate,
  File "/app/.heroku/python/lib/python3.6/site-packages/amqp/channel.py", line 1732, in _basic_publish
    (0, exchange, routing_key, mandatory, immediate), msg
  File "/app/.heroku/python/lib/python3.6/site-packages/amqp/abstract_channel.py", line 50, in send_method
    conn.frame_writer(1, self.channel_id, sig, args, content)
  File "/app/.heroku/python/lib/python3.6/site-packages/amqp/method_framing.py", line 166, in write_frame
    write(view[:offset])
  File "/app/.heroku/python/lib/python3.6/site-packages/amqp/transport.py", line 275, in write
    self._write(s)
ConnectionResetError: [Errno 104] Connection reset by peer
[2018-06-25 02:46:10,375: INFO/MainProcess] Connected to amqp://***:**@***:***/***
[2018-06-25 02:46:10,526: INFO/MainProcess] mingle: searching for neighbors
[2018-06-25 02:46:11,764: INFO/MainProcess] mingle: all alone
```
Although I checked the logs, I found a log of reconnection at the timing of memory leak, but there was also a case where a memory leak started at the timing when reconnection did not occur.
I agree with the idea of jxlton.

Also, when I was using Celery 3.x, I did not encounter such a problem.
same problem here
<img width="802" alt="screenshot 2018-06-25 11 09 22" src="https://user-images.githubusercontent.com/1920678/41831344-48386766-7868-11e8-88bd-a2918fc43369.png">
Every few days i have to restart workers due to this problem
there are no any significant clues in logs, but I have a suspicion that reconnects can affect; since i have reconnect log entries somewhere in time when memory starts constantly growing
My conf is ubuntu 17, 1 server - 1 worker with 3 concurrency; rabbit and redis on backend; all packages are the latest versions
@marvelph @dmitry-kostin could you please provide your exact configuration (omitting sensitive information of course) and possibly a task, or sample, that reproduces the issue? Also, do you have any estimate of the average uptime interval that the worker memory increase starts appearing?
the config is nearby to default

imports = ('app.tasks',)
result_persistent = True
task_ignore_result = False
task_acks_late = True
worker_concurrency = 3
worker_prefetch_multiplier = 4
enable_utc = True
timezone = 'Europe/Moscow'
broker_transport_options = {'visibility_timeout': 3600, 'confirm_publish': True, 'fanout_prefix': True, 'fanout_patterns': True}

<img width="777" alt="screenshot 2018-06-25 11 35 17" src="https://user-images.githubusercontent.com/1920678/41832003-e0ef4c74-786b-11e8-9591-7c75453f7f29.png">
Basically this is new deployed node; it was deployed on 06/21 18-50; stared to grow 6/23 around 05-00 and finally crashed 6/23 around 23-00

the task is pretty simple and there is no superlogic there, i think i can reproduce the whole situation on a clear temp project but have no free time for now, if i will be lucky i will try to do a full example on weekend

UPD
as you can see the task itself consumes some memory you can see it by spikes on the graph, but the time when memory stared to leak there were no any tasks produced or any other activities 

@marvelph @dmitry-kostin @jxltom I noticed you use Python3. Would you mind enabling [tracemalloc](https://docs.python.org/3/library/tracemalloc.html) for the process? You may need to patch the worker process though to log memory allocation traces, let me know if you need help with that.
@georgepsarakis You mean enable tracemalloc in worker and log stats, such as the top 10 memory usage files, at a specific interval such as 5 minutes?
@jxltom I think something like that would help locate the part of code that is responsible. What do you think?
@georgepsarakis I'v tried to use gdb and https://github.com/lmacken/pyrasite to inject the memory leak process, and start debug via tracemalloc. Here is the top 10 file with highest mem usage.

I use ```resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024``` and the memory usage is gradually increasing indeed.

```
>>> import tracemalloc
>>> 
>>> tracemalloc.start()
>>> snapshot = tracemalloc.take_snapshot()
>>> top_stats = snapshot.statistics('lineno')
>>> for stat in top_stats[:10]:
...     print(stat)
... 
/app/.heroku/python/lib/python3.6/site-packages/kombu/utils/eventio.py:84: size=12.0 KiB, count=1, average=12.0 KiB
/app/.heroku/python/lib/python3.6/site-packages/celery/worker/heartbeat.py:47: size=3520 B, count=8, average=440 B
/app/.heroku/python/lib/python3.6/site-packages/amqp/method_framing.py:166: size=3264 B, count=12, average=272 B
/app/.heroku/python/lib/python3.6/site-packages/celery/events/dispatcher.py:142: size=3060 B, count=10, average=306 B
/app/.heroku/python/lib/python3.6/site-packages/celery/events/dispatcher.py:157: size=2912 B, count=8, average=364 B
/app/.heroku/python/lib/python3.6/site-packages/amqp/abstract_channel.py:50: size=2912 B, count=8, average=364 B
/app/.heroku/python/lib/python3.6/site-packages/kombu/messaging.py:181: size=2816 B, count=12, average=235 B
/app/.heroku/python/lib/python3.6/site-packages/kombu/messaging.py:203: size=2816 B, count=8, average=352 B
/app/.heroku/python/lib/python3.6/site-packages/celery/events/dispatcher.py:199: size=2672 B, count=6, average=445 B
/app/.heroku/python/lib/python3.6/site-packages/amqp/channel.py:1734: size=2592 B, count=8, average=324 B
```

Here is the difference between two snapshots after around 5 minutes.

```
>>> snapshot2 = tracemalloc.take_snapshot()
>>> top_stats = snapshot2.compare_to(snapshot, 'lineno')
>>> print("[ Top 10 differences ]")
[ Top 10 differences ]

>>> for stat in top_stats[:10]:
...     print(stat)
... 
/app/.heroku/python/lib/python3.6/site-packages/celery/worker/heartbeat.py:47: size=220 KiB (+216 KiB), count=513 (+505), average=439 B
/app/.heroku/python/lib/python3.6/site-packages/celery/events/dispatcher.py:142: size=211 KiB (+208 KiB), count=758 (+748), average=285 B
/app/.heroku/python/lib/python3.6/site-packages/amqp/method_framing.py:166: size=210 KiB (+206 KiB), count=789 (+777), average=272 B
/app/.heroku/python/lib/python3.6/site-packages/celery/events/dispatcher.py:157: size=190 KiB (+187 KiB), count=530 (+522), average=366 B
/app/.heroku/python/lib/python3.6/site-packages/amqp/abstract_channel.py:50: size=186 KiB (+183 KiB), count=524 (+516), average=363 B
/app/.heroku/python/lib/python3.6/site-packages/celery/events/dispatcher.py:199: size=185 KiB (+182 KiB), count=490 (+484), average=386 B
/app/.heroku/python/lib/python3.6/site-packages/kombu/messaging.py:203: size=182 KiB (+179 KiB), count=528 (+520), average=353 B
/app/.heroku/python/lib/python3.6/site-packages/kombu/messaging.py:181: size=179 KiB (+176 KiB), count=786 (+774), average=233 B
/app/.heroku/python/lib/python3.6/site-packages/amqp/channel.py:1734: size=165 KiB (+163 KiB), count=525 (+517), average=323 B
/app/.heroku/python/lib/python3.6/site-packages/kombu/async/hub.py:293: size=157 KiB (+155 KiB), count=255 (+251), average=632 B
```
Any suggestions for how to continue to debug this? I have no clue for how to proceed. Thanks.
@georgepsarakis

I want a little time to cut out the project for reproduction.

It is setting of Celery.

```
BROKER_URL = [
    'amqp://xxxxxxxx:yyyyyyyy@aaa.bbb.ccc.ddd:5672/zzzzzzzz'
]
BROKER_TRANSPORT_OPTIONS = {}
```

The scheduler has the following settings.

```
CELERYBEAT_SCHEDULE = {
    'aaaaaaaa_bbbbbbbb': {
        'task': 'aaaa.bbbbbbbb_cccccccc',
        'schedule': celery.schedules.crontab(minute=0),
    },
    'dddddddd_eeeeeeee': {
        'task': 'dddd.eeeeeeee_ffffffff',
        'schedule': celery.schedules.crontab(minute=0),
    },
}
```

On EC 2, I am using supervisord to operate it.
@georgepsarakis
Since my test environment can tolerate performance degradation, you can use tracemalloc.
Can you make a patched Celery to dump memory usage?
@jxltom  I bet tracemalloc with 5 minutes wont help to locate problem
For example I have 5 nodes and only 3 of them had this problem for last 4 days, and 2 worked fine all this this time, so it will be very tricky to locate problem .. 
I feel like there is some toggle that switches on and then memory starts grow, until this switch memory consumption looks very well 
I tried to find out whether similar problems occurred in other running systems.
The frequency of occurrence varies, but a memory leak has occurred on three systems using Celery 4.x, and it has not happened on one system.
The system that has a memory leak is Python 3.5.x, and the system with no memory leak is Python 2.7.x.
@dmitry-kostin What's the difference with the other two normal nodes, are they both using same rabbitmq as broker?

Since our discussion mentioned it may related to rabbitmq, I started another new node with same configuration except for using redis instead. So far, this node has no memory leak after running 24 hours. I will post it here if it has memory leak later
@marvelph So do you mean that the three system with memory leak are using python3 while the one which is fine is using python2?
@jxltom no difference at all, and yes they are on python 3 & rabit as broker and redis on backend
I made a testing example to reproduce this, if it will succeed in a couple of days i will give credentials to this servers for somebody who aware how to locate this bug
@jxltom
Yes.
As far as my environment is concerned, problems do not occur in Python 2.
I tracked the memory leak via tracemalloc within a longer period.

The start memory usage reported by ```resource``` module is ```146.58MB```, after the 3.5 hours, it reports the memory usage is ```224.21MB```.

Following is the snapshot difference reported by ```tracemalloc```

``` 
>>> snapshot2 = tracemalloc.take_snapshot(); top_stats = snapshot2.compare_to(snapshot, 'lineno')
>>> for stat in top_stats[:10]:
...     print(stat)
... 
/app/.heroku/python/lib/python3.6/site-packages/celery/worker/heartbeat.py:47: size=3619 KiB (+3614 KiB), count=8436 (+8426), average=439 B
/app/.heroku/python/lib/python3.6/site-packages/celery/events/dispatcher.py:142: size=3470 KiB (+3466 KiB), count=12529 (+12514), average=284 B
/app/.heroku/python/lib/python3.6/site-packages/amqp/method_framing.py:166: size=3418 KiB (+3414 KiB), count=12920 (+12905), average=271 B
/app/.heroku/python/lib/python3.6/site-packages/celery/events/dispatcher.py:157: size=3149 KiB (+3145 KiB), count=8762 (+8752), average=368 B
/app/.heroku/python/lib/python3.6/site-packages/amqp/abstract_channel.py:50: size=3099 KiB (+3096 KiB), count=8685 (+8676), average=365 B
/app/.heroku/python/lib/python3.6/site-packages/celery/events/dispatcher.py:199: size=3077 KiB (+3074 KiB), count=8354 (+8345), average=377 B
/app/.heroku/python/lib/python3.6/site-packages/kombu/messaging.py:203: size=3020 KiB (+3017 KiB), count=8723 (+8713), average=355 B
/app/.heroku/python/lib/python3.6/site-packages/kombu/messaging.py:181: size=2962 KiB (+2959 KiB), count=12952 (+12937), average=234 B
/app/.heroku/python/lib/python3.6/site-packages/amqp/channel.py:1734: size=2722 KiB (+2718 KiB), count=8623 (+8613), average=323 B
/app/.heroku/python/lib/python3.6/site-packages/kombu/async/hub.py:293: size=2588 KiB (+2585 KiB), count=4193 (+4188), average=632 B
```

Any ideas? It looks like it is not a single file is leaking?

I also imported ```gc```, and ```gc.collect()``` returns ```0```...
@georgepsarakis I was able to reproduce this, ping me for access creds
Update: I have updated the broker from rabbitmq to redis by updating broker url as environment variable and keep docker/code completely same. And it's running for 4 days and **there is no memory leak**.

So I believe this issue is related with rabbitmq broker.
If possible please try running the benchmark command, mentioned here: https://github.com/celery/celery/issues/2927#issuecomment-171455414
This system is running workers with 20 servers.
A memory leak occurred yesterday, but it is occurring on almost all servers at the same time.
![memoryleak](https://user-images.githubusercontent.com/280983/42120084-3521f930-7c50-11e8-91c1-965237b21804.jpeg)

Don't know if it's related, leaving it here in case it helps.

I have a different issue with celery and rabbitmq (celery loses connection and starts reconnecting loads of times per second, cpu goes 100% on 1 core, beat can't send new tasks, need to restart celery).

The reason I am reporting this here is the trigger: after days of monitoring I think I located the start of the issue and it appears to be rabbitmq moving some messages from memory to disk. At that time celery starts trying to reconnect as fast as it can and rabbitmq logs show tens of connection/disconnection operations per second, in batches of ~10 or so at a time. Restarting rabbitmq doesn't fix the issue, restarting celery fixes it right away. I do not have a proper fix but as an example, setting an expire policy allowing messages to always stay in memory works around the issue and I haven't seen it since.

Given some details of this issue match what I saw (swapping rabbitmq with redis fixes it, there's not a clear starting point, it happens on more than one worker/server at the same time) I guess there might be a common trigger and it might be the same I spotted.
The test suite is changed from ```https://github.com/celery/celery/tree/master/funtests/stress``` to ```https://github.com/celery/cyanide```, and it only supports Python2.

So I run it in Python2 with rabbitmq as broker. It raised ```!join: connection lost: error(104, 'Connection reset by peer')```. Is this related with memory leak issue?

Here is the log for test suite.

```
➜  cyanide git:(master) pipenv run python -m cyanide.bin.cyanide
Loading .env environment variables…
Cyanide v1.3.0 [celery 4.2.0 (windowlicker)]

Linux-4.13.0-45-generic-x86_64-with-debian-stretch-sid

[config]
.> app:    cyanide:0x7fb097f31710
.> broker: amqp://**:**@**:**/cyanide
.> suite: cyanide.suites.default:Default

[toc: 12 tests total]
.> 1) manyshort,
.> 2) always_timeout,
.> 3) termbysig,
.> 4) timelimits,
.> 5) timelimits_soft,
.> 6) alwayskilled,
.> 7) alwaysexits,
.> 8) bigtasksbigvalue,
.> 9) bigtasks,
.> 10) smalltasks,
.> 11) revoketermfast,
.> 12) revoketermslow

+enable worker task events...
+suite start (repetition 1)
[[[manyshort(50)]]]
 1: manyshort                            OK (1/50) rep#1 runtime: 15.00 seconds/15.01 seconds
 1: manyshort                            OK (2/50) rep#1 runtime: 13.16 seconds/28.17 seconds
 1: manyshort                            OK (3/50) rep#1 runtime: 13.29 seconds/41.46 seconds
 1: manyshort                            OK (4/50) rep#1 runtime: 13.70 seconds/55.16 seconds
 1: manyshort                            OK (5/50) rep#1 runtime: 13.77 seconds/1.15 minutes
 1: manyshort                            OK (6/50) rep#1 runtime: 13.91 seconds/1.38 minutes
!join: connection lost: error(104, 'Connection reset by peer')
!Still waiting for 1000/1000: [1624cd7a-3cc0-474a-b957-b0484f6b4937, 2b436525-73de-4062-bd6b-924cbd11ba74, ce04cb5e-a99e-41e2-95dc-e9bc351e606d,...]: TimeoutError(u'The operation timed out.',)
!Still waiting for 1000/1000: [1624cd7a-3cc0-474a-b957-b0484f6b4937, 2b436525-73de-4062-bd6b-924cbd11ba74, ce04cb5e-a99e-41e2-95dc-e9bc351e606d,...]: TimeoutError(u'The operation timed out.',)
!Still waiting for 475/1000: [1624cd7a-3cc0-474a-b957-b0484f6b4937, 2b436525-73de-4062-bd6b-924cbd11ba74, ce04cb5e-a99e-41e2-95dc-e9bc351e606d,...]: TimeoutError(u'The operation timed out.',)
!Still waiting for 1000/1000: [1624cd7a-3cc0-474a-b957-b0484f6b4937, 2b436525-73de-4062-bd6b-924cbd11ba74, ce04cb5e-a99e-41e2-95dc-e9bc351e606d,...]: TimeoutError(u'The operation timed out.',)
!Still waiting for 1000/1000: [1624cd7a-3cc0-474a-b957-b0484f6b4937, 2b436525-73de-4062-bd6b-924cbd11ba74, ce04cb5e-a99e-41e2-95dc-e9bc351e606d,...]: TimeoutError(u'The operation timed out.',)
!Still waiting for 475/1000: [1624cd7a-3cc0-474a-b957-b0484f6b4937, 2b436525-73de-4062-bd6b-924cbd11ba74, ce04cb5e-a99e-41e2-95dc-e9bc351e606d,...]: TimeoutError(u'The operation timed out.',)
!Still waiting for 1000/1000: [1624cd7a-3cc0-474a-b957-b0484f6b4937, 2b436525-73de-4062-bd6b-924cbd11ba74, ce04cb5e-a99e-41e2-95dc-e9bc351e606d,...]: TimeoutError(u'The operation timed out.',)
!Still waiting for 1000/1000: [1624cd7a-3cc0-474a-b957-b0484f6b4937, 2b436525-73de-4062-bd6b-924cbd11ba74, ce04cb5e-a99e-41e2-95dc-e9bc351e606d,...]: TimeoutError(u'The operation timed out.',)
!join: connection lost: error(104, 'Connection reset by peer')
failed after 7 iterations in 3.12 minutes
Traceback (most recent call last):
  File "/home/***/.pyenv/versions/2.7.15/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/***/.pyenv/versions/2.7.15/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/***/Documents/Python-Dev/cyanide/cyanide/bin/cyanide.py", line 62, in <module>
    main()
  File "/home/***/Documents/Python-Dev/cyanide/cyanide/bin/cyanide.py", line 58, in main
    return cyanide().execute_from_commandline(argv=argv)
  File "/home/***/.local/share/virtualenvs/cyanide-Vy3PQPTU/lib/python2.7/site-packages/celery/bin/base.py", line 275, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/home/***/.local/share/virtualenvs/cyanide-Vy3PQPTU/lib/python2.7/site-packages/celery/bin/base.py", line 363, in handle_argv
    return self(*args, **options)
  File "/home/***/.local/share/virtualenvs/cyanide-Vy3PQPTU/lib/python2.7/site-packages/celery/bin/base.py", line 238, in __call__
    ret = self.run(*args, **kwargs)
  File "/home/***/Documents/Python-Dev/cyanide/cyanide/bin/cyanide.py", line 20, in run
    return self.run_suite(names, **options)
  File "/home/***/Documents/Python-Dev/cyanide/cyanide/bin/cyanide.py", line 30, in run_suite
    ).run(names, **options)
  File "cyanide/suite.py", line 366, in run
    self.runtest(test, iterations, j + 1, i + 1)
  File "cyanide/suite.py", line 426, in runtest
    self.execute_test(fun)
  File "cyanide/suite.py", line 447, in execute_test
    fun()
  File "cyanide/suites/default.py", line 22, in manyshort
    timeout=10, propagate=True)
  File "cyanide/suite.py", line 246, in join
    raise self.TaskPredicate('Test failed: Missing task results')
cyanide.suite.StopSuite: Test failed: Missing task results
```

Here is the log for worker.

```
➜  cyanide git:(master) pipenv run celery -A cyanide worker -c 1
Loading .env environment variables…
 
 -------------- celery@** v4.2.0 (windowlicker)
---- **** ----- 
--- * ***  * -- Linux-4.13.0-45-generic-x86_64-with-debian-stretch-sid 2018-07-03 12:59:28
-- * - **** --- 
- ** ---------- [config]
- ** ---------- .> app:         cyanide:0x7fdc988b4e90
- ** ---------- .> transport:   amqp://**:**@**:**/cyanide
- ** ---------- .> results:     rpc://
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> c.stress         exchange=c.stress(direct) key=c.stress
                

[2018-07-03 12:59:29,883: WARNING/ForkPoolWorker-1] ! Still waiting for 1000/1000: [e6e71bed-8e58-4e7e-96c5-f56b583a37af, 42fd4f97-4ff5-4e0e-b874-89e7b3f0ff22, 3de45eeb-9b89-41bc-8284-95a4c07aa34a,...]: TimeoutError('The operation timed out.',) !
[2018-07-03 12:59:29,886: WARNING/ForkPoolWorker-1] ! Still waiting for 1000/1000: [e6e71bed-8e58-4e7e-96c5-f56b583a37af, 42fd4f97-4ff5-4e0e-b874-89e7b3f0ff22, 3de45eeb-9b89-41bc-8284-95a4c07aa34a,...]: TimeoutError('The operation timed out.',) !
[2018-07-03 12:59:30,964: WARNING/ForkPoolWorker-1] + suite start (repetition 1) +
[2018-07-03 12:59:30,975: WARNING/ForkPoolWorker-1] ---  1: manyshort                             (0/50) rep#1 runtime: 0.0000/0.0000 ---
[2018-07-03 13:01:07,835: WARNING/ForkPoolWorker-1] ! join: connection lost: error(104, 'Connection reset by peer') !
[2018-07-03 13:01:17,918: WARNING/ForkPoolWorker-1] ! Still waiting for 1000/1000: [1624cd7a-3cc0-474a-b957-b0484f6b4937, 2b436525-73de-4062-bd6b-924cbd11ba74, ce04cb5e-a99e-41e2-95dc-e9bc351e606d,...]: TimeoutError(u'The operation timed out.',) !
[2018-07-03 13:01:27,951: WARNING/ForkPoolWorker-1] ! Still waiting for 1000/1000: [1624cd7a-3cc0-474a-b957-b0484f6b4937, 2b436525-73de-4062-bd6b-924cbd11ba74, ce04cb5e-a99e-41e2-95dc-e9bc351e606d,...]: TimeoutError(u'The operation timed out.',) !
[2018-07-03 13:01:38,902: WARNING/ForkPoolWorker-1] ! Still waiting for 475/1000: [1624cd7a-3cc0-474a-b957-b0484f6b4937, 2b436525-73de-4062-bd6b-924cbd11ba74, ce04cb5e-a99e-41e2-95dc-e9bc351e606d,...]: TimeoutError(u'The operation timed out.',) !
[2018-07-03 13:01:48,934: WARNING/ForkPoolWorker-1] ! Still waiting for 1000/1000: [1624cd7a-3cc0-474a-b957-b0484f6b4937, 2b436525-73de-4062-bd6b-924cbd11ba74, ce04cb5e-a99e-41e2-95dc-e9bc351e606d,...]: TimeoutError(u'The operation timed out.',) !
[2018-07-03 13:01:58,961: WARNING/ForkPoolWorker-1] ! Still waiting for 1000/1000: [1624cd7a-3cc0-474a-b957-b0484f6b4937, 2b436525-73de-4062-bd6b-924cbd11ba74, ce04cb5e-a99e-41e2-95dc-e9bc351e606d,...]: TimeoutError(u'The operation timed out.',) !
[2018-07-03 13:02:09,906: WARNING/ForkPoolWorker-1] ! Still waiting for 475/1000: [1624cd7a-3cc0-474a-b957-b0484f6b4937, 2b436525-73de-4062-bd6b-924cbd11ba74, ce04cb5e-a99e-41e2-95dc-e9bc351e606d,...]: TimeoutError(u'The operation timed out.',) !
[2018-07-03 13:02:19,934: WARNING/ForkPoolWorker-1] ! Still waiting for 1000/1000: [1624cd7a-3cc0-474a-b957-b0484f6b4937, 2b436525-73de-4062-bd6b-924cbd11ba74, ce04cb5e-a99e-41e2-95dc-e9bc351e606d,...]: TimeoutError(u'The operation timed out.',) !
[2018-07-03 13:02:29,964: WARNING/ForkPoolWorker-1] ! Still waiting for 1000/1000: [1624cd7a-3cc0-474a-b957-b0484f6b4937, 2b436525-73de-4062-bd6b-924cbd11ba74, ce04cb5e-a99e-41e2-95dc-e9bc351e606d,...]: TimeoutError(u'The operation timed out.',) !
[2018-07-03 13:02:37,900: WARNING/ForkPoolWorker-1] ! join: connection lost: error(104, 'Connection reset by peer') !
```
I have updated to use celery 3.1.25 with same stress test suite, everything is fine.
BTW For everybody who are looking for a fast fix – replacing rabbit with redis solves the problem as @jxltom suggested, i have more than week of stable work with redis only now
So the problem is definitely somewhere near the rabbit<->celery border
@dieeasy we have experienced the same issue. I assume you are using RPC result backend. If so, try switching to DB result backend and see if that helps. The issue that causes this is: https://github.com/celery/kombu/pull/779 and is explained here: https://github.com/celery/kombu/pull/779#discussion_r134961611
I have a same problem memory leak
**Memory**
![image](https://user-images.githubusercontent.com/17918689/48121683-8c412080-e2b0-11e8-9c31-7d6819c212c6.png)

**Version**
python `3.6.5`  celery `4.2.1`   backend `redis`   broker `rabbitmq`

**Config**
```conf
[celery]
broker_url=amqp://taunt:xxxx@xx.xx.xx.xx:5672/%2ftaunt
celery_result_backend=redis://xx.xx.xx.xx:6379
# 7days
celery_task_result_expires=604800
celery_task_serializer=msgpack
celery_result_serializer=json
celery_accept_content=json,msgpack
celery_timezone=Asia/Shanghai
celery_enable_utc=True

[cmd]
worker=True
proj=app.worker.celery
log_level=INFO
name=send_im%%h
queue=im
autoscale=10,3
concurrency=10

```
```python
# -*- coding: utf-8 -*-

from kombu import Queue, Exchange
from oslo_log import log as logging

from app.conf import CONF

LOG = logging.getLogger(__name__)


celery_queues = (
    Queue('im', exchange=Exchange('sender'), routing_key='im'),
    Queue('sms', exchange=Exchange('sender'), routing_key='sms'),
    Queue('mail', exchange=Exchange('sender'), routing_key='mail'),
    Queue('ivr', exchange=Exchange('sender'), routing_key='ivr')
)


celery_routes = {
    'sender.im': {'queue': 'im', 'routing_key': 'im'},
    'sender.sms': {'queue': 'sms', 'routing_key': 'sms'},
    'sender.mail': {'queue': 'mail', 'routing_key': 'mail'},
    'sender.ivr': {'queue': 'ivr', 'routing_key': 'ivr'}
}


config = {
    'BROKER_URL': CONF.celery.broker_url,
    'CELERY_RESULT_BACKEND': CONF.celery.celery_result_backend,
    'CELERY_TASK_RESULT_EXPIRES': CONF.celery.celery_task_result_expires,
    'CELERY_TASK_SERIALIZER': CONF.celery.celery_task_serializer,
    'CELERY_RESULT_SERIALIZER': CONF.celery.celery_result_serializer,
    'CELERY_ACCEPT_CONTENT': CONF.celery.celery_accept_content.split(','),
    'CELERY_TIMEZONE': CONF.celery.celery_timezone,
    'CELERY_ENABLE_UTC': CONF.celery.celery_enable_utc,
    'CELERY_QUEUES': celery_queues,
    'CELERY_ROUTES': celery_routes
}

```

**Startup**
```python

def make_command() -> list:
    log_path = f'{CONF.log_dir}{os.sep}{CONF.log_file}'
    command_name = f'{sys.path[0]}{os.sep}celery'
    command = [command_name, 'worker', '-A', CONF.cmd.proj, '-E']
    if CONF.cmd.log_level:
        command.extend(['-l', CONF.cmd.log_level])
    if CONF.cmd.queue:
        command.extend(['-Q', CONF.cmd.queue])
    if CONF.cmd.name:
        command.extend(['-n', CONF.cmd.name])
    # if CONF.cmd.autoscale:
    #     command.extend(['--autoscale', CONF.cmd.autoscale])
    if CONF.cmd.concurrency:
        command.extend(['--concurrency', CONF.cmd.concurrency])
    command.extend(['-f', log_path]) 
    return command


if CONF.cmd.worker:
    LOG.info(make_command())
    entrypoint = celery.start(argv=make_command())

```

**I can provide more information if needed.**
For what it's worth, I am having this issue and can reproduce it consistently by opening the rabbitmq management console, going to connections, and closing connections with traffic from celery to rabbitmq.

I've tested with celery 4.1 and 4.2 and rabbitmq 3.7.7-1
EDIT: also python version 3.6.5 and the ubuntu 16.04 (AWS EC2 image)
I'm having a memory leak with celery 4.2.1 and redis broker. The memory grows from 100 MiB to 500 MiB(limited) in 3 hours, and the workers are marked as offline in flower. Both prefork pool and gevent show the same issue.
@yifeikong this may not be the same issue, but for your case could you please try the solution proposed https://github.com/celery/celery/pull/4839#issuecomment-447739820 ?
@georgepsarakis I'm using Python 3.6.5, so I'm not affected by this bug. I will use tracemalloc to do some research. If it was a celery bug, I'll open a new issue. Thanks
Maybe same cause with [#5047](https://github.com/celery/celery/issues/5047), it seems this bug can lead to different phenomenon.
We are facing the same memory leak running Celery 4.2.1, Kombu 4.2.2 and python3.6 with RabbitMQ as broker. 

```
$ celery --app=eventr.celery_app report

software -> celery:4.2.1 (windowlicker) kombu:4.2.2-post1 py:3.6.8
            billiard:3.5.0.5 py-amqp:2.4.0
platform -> system:Linux arch:64bit imp:CPython
```

I can say we have tried many things that other people mentioned as possible workarounds (redis as broker, using jemalloc, libamqp, monkey path `__del__` on `AsyncResult`) but we always ended up having memory leaked. 

By analysing our log we noticed that we had a lot of messages related to missed heartbeats from gossip.

```
{"asctime": "2019-01-25 13:40:06,486", "levelname": "INFO", "name": "celery.worker.consumer.gossip", "funcName": "on_node_lost", "lineno": 147, "message": "missed heartbeat from celery@******"}
```
One last thing that we tried was disabling gossip by running the workers with `--without-gossip`, surprisingly, disabling gossip had an immediate effect.

You can see it here:
![celery-memory-14d](https://user-images.githubusercontent.com/924245/52136177-c61f8a00-2647-11e9-938f-d37c1e33455f.png)
  
Since we deactivated gossip in two projects running celery workers the memory consumption has improved.

If you pay attention, before we were having similar memory spikes as described here https://github.com/celery/celery/issues/4843#issuecomment-399833781 

One thing that I've been trying to fully understand is what are the implications of completely disabling gossip, since it's only described as worker <-> worker communication, if anyone could shed some light about this I would be very grateful.

Hope this helps and thanks for the hard work.
Why was this issue closed?
There is active feedback and interest in this issue, so I am reopening.
Well @georgepsarakis since we diagnosed my leak as not being #4839, and you [suspected](https://github.com/celery/celery/pull/4839#issuecomment-458817459) that it was #4843, I'll flip over to this leak thread at least for now.  I'm not sure #4843 is my leak either.  According to the initial issue on this thread:

> This problem happens at least in Celery 4.1, and it also occurs in Celery 4.2.
Celery is running on Ubuntu 16 and brokers use RabbitMQ.

I'm currently on:

python 2.7.12
Ubuntu 16.04.1 amd64
RabbitMQ 3.7.5

using:

Celery 4.1.1
librabbitmq 2.0.0
amqp 2.4.0
vine 1.1.4
billiard 3.5.0.5
kombu 4.2.2.post1
gevent 1.2.2

However, Celery 4.1.1 + gevent 1.2.2 doesn't leak for me (nor does Celery 3.1.25 + gevent 1.2.2 AFAICT); Celery 4.2.1 + gevent 1.3.7 does.  Unfortunately, gevent 1.3.7 and gevent 1.2.2 are not interchangeable to demonstrate (or exclude) a gevent library as a possible source of the problem.
 
EDIT:  Hmm...there seems to be a gevent patch (022f447dd) that looks like it could fix the error I encountered.  I'll try and get that to work.
I applied 022f447 to Celery 4.1.1 and installed gevent 1.3.7.  That Celery + gevent combination ran...and produced memory usage patterns consistent with the leak I've been experiencing.  I'll install Celery 4.2.1 + gevent 1.2.2 (with the reverse patch) and see if I get the usual memory usage pattern.

I notice gevent 1.4.0 is out.  Maybe I should give that a whirl as well to see how that behaves.
Celery 4.2.1 + gevent 1.2.2 + reverse patch for gevent 1.2.2 doesn't seem to produce the leak as does Celery 4.2.1 + gevent 1.3.7. 
Celery 4.2.1 + gevent 1.4.0 does seem to leak at approximately the same rate as gevent 1.3.7 AFAICT.
https://github.com/celery/celery/blob/9f0a554dc2d28c630caf9d192873d040043b7346/celery/events/dispatcher.py
```python
    def _publish(self, event, producer, routing_key, retry=False,
                 retry_policy=None, utcoffset=utcoffset):
        exchange = self.exchange
        try:
            producer.publish(...)
        except Exception as exc:  # pylint: disable=broad-except
            if not self.buffer_while_offline:  # <-- False by default
                raise
            self._outbound_buffer.append((event, routing_key, exc))  # <---- Always buffered

    def send(self, type, blind=False, utcoffset=utcoffset, retry=False,
            ...
            if group in self.buffer_group:   # <--- Never true for eventlet & gevent
                ...
                if len(buf) >= self.buffer_limit:
                    self.flush()     #  <---- Never flushed even when grows above limit
                ...
            else:
                return self.publish(type, fields, self.producer, blind=blind,
                                    Event=Event, retry=retry,
```
https://github.com/celery/celery/blob/b2668607c909c61becd151905b4525190c19ff4a/celery/worker/consumer/events.py
```python
    def start(self, c):
        # flush events sent while connection was down.
        prev = self._close(c)
        dis = c.event_dispatcher = c.app.events.Dispatcher(
            ...
            # we currently only buffer events when the event loop is enabled
            # XXX This excludes eventlet/gevent, which should actually buffer.
            buffer_group=['task'] if c.hub else None,
            on_send_buffered=c.on_send_event_buffered if c.hub else None,
        )
        if prev:
            dis.extend_buffer(prev)
            dis.flush()    # <---- The only (!) chance to flush on [g]event[let] is on reconnect.
```
Now, if I understand correctly what AMQP does under the hood, then it has it's own heartbeat and when it detects a broken connection, it goes ahead and reconnects under the hood. Depending on the types of events that are enabled (gossip, heartbeat), this can leak pretty fast.
This should be true for any version of eventlet & gevent but some could exhibit connection issues that make things worse/more noticeable.
Hi,

I suspect that we are having the same issue.
Our configuration is below. Can I either negate or confirm that this is the same issue discussed here?

Python: 2.7
Celery: 4.2.1
OS: CentOS release 6.10
Redis as broker

In the attached image you can see:

1. Memory consumption increasing constantly and dropping on restart.
2. On January 13 - we upgraded from celery 3.1.25 to 4.2.1. Memory consumption increasing pace grows.

![image](https://user-images.githubusercontent.com/41108188/52936528-18a9b780-3365-11e9-93dd-2fd076f52da7.png)


**UPDATE**

Regardless this issue, we upgraded to python 3.6 and since then it seems like the leak does not happen anymore.

![image](https://user-images.githubusercontent.com/41108188/54082547-13b2a500-4320-11e9-810d-4f1a2f3c9e60.png)
(the upgrade was on February 19)

@georgepsarakis 
Not sure how relevant this is, but I'm having my 2GB of SWAP space exhausted by celery in production. Stopping Flower didn't clear the memory, but stopping Celery did.
could anyone try celery 4.3rc1?
@auvipy I installed Celery 4.3.0rc1 + gevent 1.4.0.  pip upgraded billiard to 3.6.0.0 and kombu 4.3.0.

Kind of puzzled that vine 1.2.0 wasn't also required by the rc1 package, given that #4839 is fixed by that upgrade.

Anyway, Celery 4.3.0 rc1 seems to run OK.
@ldav1s thanks a lot for the feedback. So, vine is declared as a dependency in [py-amqp](https://github.com/celery/py-amqp/blob/master/requirements/default.txt) actually. In new installations the latest `vine` version will be installed but this might not happen in existing ones.

@thedrow perhaps we should declare the dependency in Celery requirements too?
Let's open an issue about it and discuss it there. 
Celery 4.3.0rc1 + gevent 1.4.0 has been running a couple of days now.   Looks like it's leaking in the same fashion as Celery 4.2.1 + gevent 1.4.0.
![image](https://user-images.githubusercontent.com/962869/53681336-d8e1b900-3cf0-11e9-9529-912617311edb.png)

Having the same leak with celery 4.2.1, python 3.6

Any updates on this? 
having same problem here
Greetings,

I'm experiencing a similar issue, but I'm not sure it is the same.

After I have migrated our celery app in a different environment/network, celery workers started to leak. Previously the celery application and the rabbitmq instance were in the same environment/network.

My configuration is on Python 3.6.5:
```
amqp (2.4.2)
billiard (3.5.0.5)
celery (4.1.1)
eventlet (0.22.0)
greenlet (0.4.15)
kombu (4.2.1)
vine (1.3.0)
```

celeryconfig

```
broker_url = rabbitmq
result_backend = mongodb
task_acks_late = True
result_expires = 0
task_default_rate_limit = 2000
task_soft_time_limit = 120
task_reject_on_worker_lost = True
loglevel = 'INFO'
worker_pool_restarts = True
broker_heartbeat = 0
broker_pool_limit = None
```

The application is composed by several workers with eventlet pool, started via command in supervisord:
```
[program:worker1]
command={{ celery_path }} worker -A celery_app --workdir {{ env_path }} -l info -E -P eventlet -c 250 -n worker1@{{ hostname }} -Q queue1,queue2
```

The memory leak behaviour it looks like this, every ~10 hours usually 1 worker, max 2 start leaking:
![image](https://user-images.githubusercontent.com/25099457/55228135-4cec7f80-5211-11e9-80ac-5a36789d21be.png)

So I have created a broadcast message for being executed on each worker, for using tracemalloc, this is the result of top command on the machine, there is 1 worker only leaking with 1464m:


```
217m   1%   2   0% /usr/bin/python3 -m celery worker -A celery_app --workdir   379
189m   1%   0   0% /usr/bin/python3 -m celery worker -A celery_app --workdir   377     
1464m   9%   1   0% /usr/bin/python3 -m celery worker -A celery_app --workdir   378
218m   1%   0   0% /usr/bin/python3 -m celery worker -A celery_app --workdir   376 
217m   1%   2   0% /usr/bin/python3 -m celery worker -A celery_app --workdir   375
217m   1%   3   0% /usr/bin/python3 -m celery worker -A celery_app --workdir   394
163m   1%   0   0% /usr/bin/python3 -m celery beat -A celery_app --workdir /app
``` 


tracemalloc TOP 10 results on the leaking worker

```
[2019-03-29 07:18:03,809: WARNING/MainProcess] [ Top 10: worker5@hostname ]

[2019-03-29 07:18:03,809: WARNING/MainProcess] /usr/lib/python3.6/site-packages/eventlet/greenio/base.py:207: size=17.7 MiB, count=26389, average=702 B

[2019-03-29 07:18:03,810: WARNING/MainProcess] /usr/lib/python3.6/site-packages/kombu/messaging.py:203: size=16.3 MiB, count=44422, average=385 B

[2019-03-29 07:18:03,811: WARNING/MainProcess] /usr/lib/python3.6/site-packages/celery/worker/heartbeat.py:49: size=15.7 MiB, count=39431, average=418 B

[2019-03-29 07:18:03,812: WARNING/MainProcess] /usr/lib/python3.6/site-packages/celery/events/dispatcher.py:156: size=13.0 MiB, count=40760, average=334 B

[2019-03-29 07:18:03,812: WARNING/MainProcess] /usr/lib/python3.6/site-packages/eventlet/greenio/base.py:363: size=12.9 MiB, count=19507, average=695 B

[2019-03-29 07:18:03,813: WARNING/MainProcess] /usr/lib/python3.6/site-packages/amqp/transport.py:256: size=12.7 MiB, count=40443, average=328 B

[2019-03-29 07:18:03,814: WARNING/MainProcess] /usr/lib/python3.6/site-packages/celery/events/dispatcher.py:138: size=12.4 MiB, count=24189, average=539 B

[2019-03-29 07:18:03,814: WARNING/MainProcess] /usr/lib/python3.6/site-packages/amqp/transport.py:256: size=12.3 MiB, count=19771, average=655 B

[2019-03-29 07:18:03,815: WARNING/MainProcess] /usr/lib/python3.6/site-packages/amqp/connection.py:505: size=11.9 MiB, count=39514, average=317 B

[2019-03-29 07:18:03,816: WARNING/MainProcess] /usr/lib/python3.6/site-packages/kombu/messaging.py:181: size=11.8 MiB, count=61362, average=201 B
```


TOP 1 with 25 frames

```
TOP 1

[2019-03-29 07:33:05,787: WARNING/MainProcess] [ TOP 1: worker5@hostname ]

[2019-03-29 07:33:05,787: WARNING/MainProcess] 26938 memory blocks: 18457.2 KiB

[2019-03-29 07:33:05,788: WARNING/MainProcess] File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 207

[2019-03-29 07:33:05,788: WARNING/MainProcess] mark_as_closed=self._mark_as_closed)

[2019-03-29 07:33:05,789: WARNING/MainProcess] File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 328

[2019-03-29 07:33:05,789: WARNING/MainProcess] timeout_exc=socket_timeout('timed out'))

[2019-03-29 07:33:05,790: WARNING/MainProcess] File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 357

[2019-03-29 07:33:05,790: WARNING/MainProcess] self._read_trampoline()

[2019-03-29 07:33:05,790: WARNING/MainProcess] File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 363

[2019-03-29 07:33:05,791: WARNING/MainProcess] return self._recv_loop(self.fd.recv, b'', bufsize, flags)

[2019-03-29 07:33:05,791: WARNING/MainProcess] File "/usr/lib/python3.6/site-packages/amqp/transport.py", line 440

[2019-03-29 07:33:05,791: WARNING/MainProcess] s = recv(n - len(rbuf))

[2019-03-29 07:33:05,792: WARNING/MainProcess] File "/usr/lib/python3.6/site-packages/amqp/transport.py", line 256

[2019-03-29 07:33:05,792: WARNING/MainProcess] frame_header = read(7, True)

[2019-03-29 07:33:05,792: WARNING/MainProcess] File "/usr/lib/python3.6/site-packages/amqp/connection.py", line 505

[2019-03-29 07:33:05,793: WARNING/MainProcess] frame = self.transport.read_frame()

[2019-03-29 07:33:05,793: WARNING/MainProcess] File "/usr/lib/python3.6/site-packages/amqp/connection.py", line 500

[2019-03-29 07:33:05,793: WARNING/MainProcess] while not self.blocking_read(timeout):

[2019-03-29 07:33:05,793: WARNING/MainProcess] File "/usr/lib/python3.6/site-packages/kombu/transport/pyamqp.py", line 103

[2019-03-29 07:33:05,794: WARNING/MainProcess] return connection.drain_events(**kwargs)

[2019-03-29 07:33:05,794: WARNING/MainProcess] File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 301

[2019-03-29 07:33:05,794: WARNING/MainProcess] return self.transport.drain_events(self.connection, **kwargs)

[2019-03-29 07:33:05,795: WARNING/MainProcess] File "/usr/lib/python3.6/site-packages/celery/worker/pidbox.py", line 120

[2019-03-29 07:33:05,795: WARNING/MainProcess] connection.drain_events(timeout=1.0)
```


I hope it could help, there are no error in the logs, other than the missed heartbeat between the workers. Now I'm trying to use the exact version of the libs we were using the old env.

UPDATE: Using the same exact dependencies lib versions and a broker heartbeat every 5 minutes the application looked like stable for longer time: more than 2 days, than it leaked again. 

There were small spike continuing for ~1hour time by time, but the were "absorbed/collected".. the last one looks like started the spike. 

After the 1st spike, 1st ramp, I have restarted the leaking worker.. as you can see another worker started to leak after it or probably it was already leaking, 2nd ramp.
 
![image](https://user-images.githubusercontent.com/25099457/55331672-a8c23d00-548b-11e9-9d22-67d7674aafa5.png)

I'm going to test without heartbeat. 

UPDATE: without heartbeat leaked again after 2 days, same behaviour 

```
440m   3%   1   0% /usr/bin/python3 -m celery worker -A celery_app --without-heartbeat --workdir /app -l info -E -P eventlet -c 250 -Ofair -n worker1@ -Q p_1_queue,p_2_queue
176m   1%   0   0% /usr/bin/python3 -m celery worker -A celery_app --without-heartbeat --workdir /app -l info -E -P eventlet -c 250 -Ofair -n worker2@ -Q p_1_queue,p_2_queue
176m   1%   2   0% /usr/bin/python3 -m celery worker -A celery_app --without-heartbeat --workdir /app -l info -E -P eventlet -c 250 -Ofair -n worker5@ -Q p_1_queue,p_2_queue
176m   1%   1   0% /usr/bin/python3 -m celery worker -A celery_app --without-heartbeat --workdir /app -l info -E -P eventlet -c 250 -Ofair -n worker3@ -Q p_1_queue,p_2_queue
176m   1%   1   0% /usr/bin/python3 -m celery worker -A celery_app --without-heartbeat --workdir /app -l info -E -P eventlet -c 250 -Ofair -n worker4@ -Q p_1_queue,p_2_queue
171m   1%   1   0% /usr/bin/python3 -m celery worker -A celery_app --without-heartbeat --workdir /app -l info -E -P eventlet -c 20 -n worker_p_root@ -Q p_root_queue
157m   1%   0   0% /usr/bin/python3 -m celery beat -A celery_app --workdir /app --schedule /app/beat.db -l info
```

![image](https://user-images.githubusercontent.com/25099457/55466927-1be8c200-55f8-11e9-84b5-f80c830a326b.png)

UPDATE:
Using celery 4.3.0 it seems the problem resolved and it is stable since a week
![image](https://user-images.githubusercontent.com/25099457/55887786-11a16780-5ba6-11e9-9209-2545dcbd8997.png)
```
amqp (2.4.2)
billiard (3.6.0.0)
celery (4.3.0)
eventlet (0.24.1)
greenlet (0.4.15)
kombu (4.5.0)
vine (1.3.0)
```



Please let me know if I could help somehow, instrumenting the code. If necessary provide links and example please.


Thank you

I'm also having a memory leak. It looks like I've managed to find the cause. 
https://github.com/celery/celery/blob/master/celery/events/dispatcher.py#L75
I can see that this buffer starts to grow after connection issues with rabbit. I don't understand why it fails to clear events eventually, it continues to grow over time and consume more and more ram. Passing `buffer_while_offline=False` here https://github.com/celery/celery/blob/master/celery/worker/consumer/events.py#L43 seems to fix the leak for me. Can someone please check if this is related?
https://github.com/celery/celery/pull/5482
@yevhen-m thank you a lot! That helped us to solve the memory leakage!
Its good that we have a workaround but can we please find a proper fix?

Resolving patch:
diff --git a/celery/events/receiver.py b/celery/events/receiver.py
--- a/celery/events/receiver.py
+++ b/celery/events/receiver.py
@@ -90,7 +90,8 @@ def capture(self, limit=None, timeout=None, wakeup=True):
         unless :attr:`EventDispatcher.should_stop` is set to True, or
         forced via :exc:`KeyboardInterrupt` or :exc:`SystemExit`.
         """
-        return list(self.consume(limit=limit, timeout=timeout, wakeup=wakeup))
+        for _ in self.consume(limit=limit, timeout=timeout, wakeup=wakeup):
+            pass
 
     def wakeup_workers(self, channel=None):
         self.app.control.broadcast('heartbeat',


Test patch: