Repository: celery/celery

Issue:
Group results aren't cleaned up on result.forget() - memory leak
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
bug reports which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->
- [x] I have verified that the issue exists against the `main` branch of Celery.
- [ ] This has already been asked to the [discussions forum](https://github.com/celery/celery/discussions) first.
- [x] I have read the relevant section in the
  [contribution guide](https://docs.celeryq.dev/en/main/contributing.html#other-bugs)
  on reporting bugs.
- [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
- [x] I have checked the [commit log](https://github.com/celery/celery/commits/main)
  to find out if the bug was already fixed in the main branch.
- [x] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).
- [x] I have tried to reproduce the issue with [pytest-celery](https://docs.celeryq.dev/projects/pytest-celery/en/latest/userguide/celery-bug-report.html) and added the reproduction script below.

## Mandatory Debugging Information

- [x] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [x] I have verified that the issue exists against the `main` branch of Celery.
- [x] I have included the contents of ``pip freeze`` in the issue.
- [x] I have included all the versions of all the external dependencies required
  to reproduce this bug.

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->
- [ ] I have tried reproducing the issue on more than one Python version
  and/or implementation.
- [ ] I have tried reproducing the issue on more than one message broker and/or
  result backend.
- [x] I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
- [x] I have tried reproducing the issue on more than one operating system.
- [ ] I have tried reproducing the issue on more than one workers pool.
- [ ] I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
- [ ] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues

- https://github.com/celery/celery/issues/9475

#### Possible Duplicates

- None

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery version**: 5.5.3 (immunity)

<!-- Include the output of celery -A proj report below -->
<details>
<summary><b><code>celery report</code> Output:</b></summary>
<p>

```
software -> celery:5.5.3 (immunity) kombu:5.5.4 py:3.13.5
            billiard:4.2.1 redis:5.2.1
platform -> system:Darwin arch:64bit, Mach-O
            kernel version:24.5.0 imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://localhost:6379/0

accept_content: ['json']
broker_url: 'redis://localhost:6379/0'
celery_settings: CelerySettings(broker_url='redis://localhost:6379/0', result_backend='redis://localhost:6379/0', result_expires=60)
enable_utc: True
result_backend: 'redis://localhost:6379/0'
result_backend_thread_safe: True
result_expires: 60
result_serializer: 'json'
task_serializer: 'json'
timezone: 'UTC'
worker_hijack_root_logger: False
include: ['data_ingest.broker.handlers']
deprecated_settings: None
```

</p>
</details>

# Steps to Reproduce

## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->
- **Minimal Python Version**: N/A or Unknown
- **Minimal Celery Version**: N/A or Unknown
- **Minimal Kombu Version**: N/A or Unknown
- **Minimal Broker Version**: N/A or Unknown
- **Minimal Result Backend Version**: N/A or Unknown
- **Minimal OS and/or Kernel Version**: N/A or Unknown
- **Minimal Broker Client Version**: N/A or Unknown
- **Minimal Result Backend Client Version**: N/A or Unknown

### Python Packages
<!-- Please fill the contents of pip freeze below -->
<details>
<summary><b><code>pip freeze</code> Output:</b></summary>
<p>

```
aiomqtt==2.4.0
amqp==5.3.1
annotated-types==0.7.0
anyio==4.9.0
billiard==4.2.1
celery==5.5.3
celery-types==0.23.0
certifi==2025.6.15
charset-normalizer==3.4.2
click==8.2.1
click-didyoumean==0.3.1
click-plugins==1.1.1.2
click-repl==0.3.0
debugpy==1.8.14
dnspython==2.7.0
docker==7.1.0
email_validator==2.2.0
fastapi==0.115.14
fastapi-cli==0.0.7
h11==0.16.0
httpcore==1.0.9
httptools==0.6.4
httpx==0.28.1
idna==3.10
influxdb==5.3.2
iniconfig==2.1.0
Jinja2==3.1.6
kombu==5.5.4
linkify-it-py==2.0.3
markdown-it-py==3.0.0
MarkupSafe==3.0.2
mdit-py-plugins==0.4.2
mdurl==0.1.2
memray==1.17.2
more-itertools==10.7.0
msgpack==1.1.1
packaging==25.0
paho-mqtt==2.1.0
platformdirs==4.3.8
pluggy==1.6.0
prompt_toolkit==3.0.51
psutil==7.0.0
pydantic==2.11.7
pydantic-settings==2.10.1
pydantic_core==2.33.2
Pygments==2.19.2
pytest==8.4.1
pytest-celery==1.2.0
pytest_docker_tools==3.1.9
python-dateutil==2.9.0.post0
python-dotenv==1.1.1
python-multipart==0.0.20
pytz==2025.2
PyYAML==6.0.2
redis==5.2.1
requests==2.32.4
rich==14.0.0
rich-toolkit==0.14.7
ruff==0.12.2
setuptools==80.9.0
shellingham==1.5.4
six==1.17.0
sniffio==1.3.1
starlette==0.46.2
structlog==25.4.0
tenacity==9.1.2
textual==3.5.0
typer==0.16.0
typing-inspection==0.4.1
typing_extensions==4.14.0
tzdata==2025.2
uc-micro-py==1.0.3
urllib3==2.5.0
uvicorn==0.34.3
uvloop==0.21.0
vine==5.1.0
watchfiles==1.1.0
wcwidth==0.2.13
websockets==15.0.1
```

</p>
</details>

### Other Dependencies
<!--
Please provide system dependencies, configuration files
and other dependency information if applicable
-->
<details>
<p>
N/A
</p>
</details>

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.

Alternatively, the pytest-celery plugin can be used to create standalone reproduction scripts
that can be added to this report. See the pytest-celery documentation for more information at
pytest-celery.readthedocs.io
-->

<details>
<p>

```python
import datetime
import logging
import random
import time

from celery import Celery, group
from celery.utils.log import get_task_logger
from pydantic import BaseModel


logging.basicConfig(level=logging.INFO)
logger = get_task_logger(__name__)


app = Celery(
    "celery_mem_leak_test",
    broker="redis://localhost:6379/0",
    backend="redis://localhost:6379/0",
    result_expires=datetime.timedelta(seconds=60),
    task_serializer="json",
)


class DataPoint(BaseModel):
    time: int
    value: float


class DataProcessorResult(BaseModel):
    data: list[DataPoint]


@app.task(acks_late=True, pydantic=True)
def first_stage(message: bytes) -> DataProcessorResult:
    lines = message.decode("utf-8").splitlines()
    data = []
    for line in lines:
        timestamp, value = line.split(",")
        data.append(DataPoint(time=int(timestamp), value=float(value)))

    return DataProcessorResult(data=data)


@app.task(acks_late=True, pydantic=True)
def group_task(result: DataProcessorResult):
    logger.info(f"Got {len(result.data)} data points")


@app.task(ignore_result=True)
def last_stage():
    logger.info("Finished task")


NUM_POINTS = 100
NUM_DISTINCT_PAYLOADS = 10
PACKET_DELAY = 1
NUM_GROUP_TASKS = 5


def main():
    # Initialize payloads
    payloads = []
    for i in range(NUM_DISTINCT_PAYLOADS):
        data = [(random.randint(0, 1000), random.random()) for _ in range(NUM_POINTS)]
        payload = "\n".join(f"{d[0]},{d[1]}" for d in data)
        payloads.append(payload.encode("utf-8"))

    while True:
        payload = random.choice(payloads)
        workflow = (
            first_stage.s(payload)
            | group(group_task.s() for _ in range(NUM_GROUP_TASKS))
            | last_stage.si()
        )
        result = workflow.apply_async()
        for mapping in result.backend._pending_results:
            logger.info(f"Number of pending_results: {len(mapping)}")
            
        # Calling get seems to fix the leak
        # try:
        #     result.get(timeout=0.1)
        # except celery.exceptions.TimeoutError:
        #     ...
        result.forget()

        time.sleep(PACKET_DELAY)


if __name__ == "__main__":
    main()

```

</p>
</details>

# Expected Behavior
<!-- Describe in detail what you expect to happen -->
I expect to observe a steady memory usage.

# Actual Behavior
<!--
Describe in detail what actually happened.
Please include a backtrace and surround it with triple backticks (```).
In addition, include the Celery daemon logs, the broker logs,
the result backend logs and system logs below if they will help us debug
the issue.
-->

I'm testing an app that submits messages to the task queue from an MQTT broker. For some reason the memory usage of both the redis broker and the app keeps steadily increasing. On the redis side the number of messages in the queue is constant, while the client output buffers keep increasing in size. I used memray to analyze the app. Added 3 outputs from running the app progressively longer each time. It's also clear that the kombu connections are using more and more memory. Will continue to debug. 

Memray graphs:
- https://gist.github.com/tsoos99dev/629ae7db4dbb0bf62228029ccc03b1aa
- https://gist.github.com/tsoos99dev/7ceff0d0aa631df93b8a452f29d2e667
- https://gist.github.com/tsoos99dev/d36393154037e0a4cd12d97632efef01

Here's the live mode output after a few hours. 

<img width="1630" height="817" alt="Image" src="https://github.com/user-attachments/assets/64e924a5-98f3-4909-8abc-48315a257084" />

<img width="1548" height="666" alt="Image" src="https://github.com/user-attachments/assets/53275fee-8a94-4455-a98b-a8dbbd28c88f" />


Discussion:
do we really needed a duplicate new issue?
> do we really needed a duplicate new issue?

Good point. The memory usage pattern is different in the first case. Still I can't say it's completely unrelated, but the apps differ. I actually removed the second and third issues I initially linked, because after further investigation I think they're unrelated. 

Alright, I found where the memory is piling up. The _pending_results instance variable of the RedisBackend instance keeps on growing, because  some results aren't cleared out after the tasks in the canvas complete. Specifically I have the following combination of tasks:
```
first_task | group(group_task1, group_task2) | last_task
```
I see 3 subscribe commands executed on redis for the following tasks:
- first_task
- group_task1
- group_task2

The worker executes the tasks, issues a bunch of commands itself, but I only see unsubscribe commands being executed for the first_task and the combined result object (We didn't even subscribe to that in the first place). The group task results stay in the _pending_results object and as more and more tasks are being generated the memory usage keeps growing. I added a simple reproduction example to the issue description. I have to add this is without using result.forget. This leaks subscriptions as well as local resources.

Maybe it's a simple misunderstanding on my end. Actually calling result.get seems to fix the issue, but since I set ignore_result=True on last_task, and I called result.forget, I didn't expect to also have to call result.get. This seems unintentional as the [docs](https://docs.celeryq.dev/en/latest/reference/celery.result.html#celery.result.AsyncResult.get) say I must eventually call get **or** forget on the result.
you can check this PR and use the idea to help reproduce it using integration tests https://github.com/celery/celery/pull/9799
I'll do that. Groups aren't usable right now. I did some more tests with calling get, and the pending task object is only cleared if the group tasks complete before the timeout. Had to completely remove groups from my app for now. 



Patch:
diff --git a/celery/result.py b/celery/result.py
index 75512c5aad..66a9e20aab 100644
--- a/celery/result.py
+++ b/celery/result.py
@@ -137,6 +137,8 @@ def forget(self):
         self._cache = None
         if self.parent:
             self.parent.forget()
+
+        self.backend.remove_pending_result(self)
         self.backend.forget(self.id)
 
     def revoke(self, connection=None, terminate=False, signal=None,


Test patch:
diff --git a/t/smoke/tests/test_canvas.py b/t/smoke/tests/test_canvas.py
index e0886d56e4..b6c69e7639 100644
--- a/t/smoke/tests/test_canvas.py
+++ b/t/smoke/tests/test_canvas.py
@@ -179,3 +179,15 @@ def test_chord_error_propagation_with_different_body_types(
         # The chord should fail with the expected exception from the failing task
         with pytest.raises(ExpectedException):
             result.get(timeout=RESULT_TIMEOUT)
+
+
+class test_complex_workflow:
+    def test_pending_tasks_released_on_forget(self, celery_setup: CeleryTestSetup):
+        sig = add.si(1, 1) | group(
+            add.s(1) | group(add.si(1, 1), add.si(2, 2)) | add.si(2, 2),
+            add.s(1) | group(add.si(1, 1), add.si(2, 2)) | add.si(2, 2)
+        ) | add.si(1, 1)
+        res = sig.apply_async(queue=celery_setup.worker.worker_queue)
+        assert not all(len(mapping) == 0 for mapping in res.backend._pending_results)
+        res.forget()
+        assert all(len(mapping) == 0 for mapping in res.backend._pending_results)
diff --git a/t/unit/tasks/test_result.py b/t/unit/tasks/test_result.py
index 062c069542..d5aaa48192 100644
--- a/t/unit/tasks/test_result.py
+++ b/t/unit/tasks/test_result.py
@@ -449,6 +449,20 @@ def test_date_done(self, utc_datetime_mock, timezone, date):
         result = Backend(app=self.app)._get_result_meta(None, states.SUCCESS, None, None)
         assert result.get('date_done') == date
 
+    def test_forget_remove_pending_result(self):
+        with patch('celery.result.AsyncResult.backend') as backend:
+            result = self.app.AsyncResult(self.task1['id'])
+            result.backend = backend
+            result_clone = copy.copy(result)
+            result.forget()
+            backend.remove_pending_result.assert_called_once_with(
+                result_clone
+            )
+
+        result = self.app.AsyncResult(self.task1['id'])
+        result.backend = None
+        del result
+
 
 class test_ResultSet: