Repository: celery/celery

Issue:
Redis Backend has a memory leak with AsyncResult.get()
## Checklist
- [X] I have included the output of ``celery -A proj report`` in the issue.
      (if you are not able to do this, then at least specify the Celery
       version affected).
```
software -> celery:4.0.2 (latentcall) kombu:4.0.2 py:3.5.2
            billiard:3.5.0.2 redis:2.10.5
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://[redacted]
```
### Celery Settings
```
BROKER_BACKEND = 'redis'
BROKER_URL = environ.get('REDIS_URL')
CELERY_RESULT_BACKEND = environ.get('REDIS_URL')
CELERYD_CONCURRENCY = 1
BROKER_POOL_LIMIT = 0
CELERY_TASK_RESULT_EXPIRES = 60
CELERY_MAX_CACHED_RESULTS = 1
CELERY_DISABLE_RATE_LIMITS = True
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
```
- [X] I have verified that the issue exists against the `master` branch of Celery.

## Steps to reproduce
1) Create celery app with Redis as transport and Backend (same cache for both)
2) Create a task within the celery app that returns a result
``` python
@CELERY_APP.task
def add():
    return 1 + 2
```
3) Execute `add.apply_async().get()` somehow (I have a Django app on Heroku so I made an endpoint for it)
4) Use `objgraph.show_growth()` to see that `AsyncResult` objects are leaking

## Expected behavior
`AsyncResult` objects get released after the result is received after the `AsyncResult.get()` call.  The `AsyncResult` is properly released when getting the result from `AsyncResult.result`.

## Actual behavior
`AsyncResult` objects accumulate in the `AsyncResult.backend._pending_results.concrete` dict.  If the `AsyncResult` is popped from the dict, it is released as expected.
Note that the leak does not occur when executing tasks eagerly.

## Code sample
``` python
@CELERY_APP.task
def add():
    return 1 + 2

def add_request():
    response = add.apply_async().get()

    import objgraph
    objgraph.show_growth()
```




Discussion:
+1 
I'm seeing leaks with AsyncResult with RPC results backend as well on Ubuntu 16.04.

```
software -> celery:4.0.2 (latentcall) kombu:4.0.2 py:2.7.12
            billiard:3.5.0.2 py-amqp:2.1.4
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:pyamqp results:rpc://admin:**@localhost/
```
Seeing this behavior as well.
@thedrow @ask [celery.result.ResultSet._on_ready](https://github.com/celery/celery/blob/master/celery/result.py#L482) no longer appears to be used anywhere, after [this change](https://github.com/celery/celery/commit/072ad1937f7d445a496369f0370033a0ba558ddf#diff-58b5816dbb95159ae9810e56eac54312L440). The promise directly calls `self.on_ready` now. Would that contribute to why the pending result is never removed?

Resolving patch:
diff --git a/celery/result.py b/celery/result.py
--- a/celery/result.py
+++ b/celery/result.py
@@ -99,7 +99,7 @@ def __init__(self, id, backend=None,
         self.id = id
         self.backend = backend or self.app.backend
         self.parent = parent
-        self.on_ready = promise(self._on_fulfilled, weak=True)
+        self.on_ready = promise(self._on_fulfilled)
         self._cache = None
 
     def then(self, callback, on_error=None, weak=False):
@@ -180,7 +180,7 @@ def get(self, timeout=None, propagate=True, interval=0.5,
             assert_will_not_block()
         _on_interval = promise()
         if follow_parents and propagate and self.parent:
-            on_interval = promise(self._maybe_reraise_parent_error, weak=True)
+            on_interval = promise(self._maybe_reraise_parent_error)
             self._maybe_reraise_parent_error()
         if on_interval:
             _on_interval.then(on_interval)
@@ -474,7 +474,7 @@ def __init__(self, results, app=None, ready_barrier=None, **kwargs):
         self.on_ready = promise(args=(self,))
         self._on_full = ready_barrier or barrier(results)
         if self._on_full:
-            self._on_full.then(promise(self.on_ready, weak=True))
+            self._on_full.then(promise(self._on_ready))
 
     def add(self, result):
         """Add :class:`AsyncResult` as a new member of the set.


Test patch: