Repository: stanfordnlp/dspy

Issue:
Memory leak in ChainOfThought module
## Issue Description

DSPy's `ChainOfThought` module exhibits a memory leak where memory continuously grows over time during usage, even when:
- Creating fresh instances for each use (no caching)
- Explicitly deleting instances after use
- Forcing garbage collection

## Impact

This causes significant production issues including:
- Kubernetes pods autoscaling unnecessarily (we've seen 3-4 pods scale to 10+ pods)
- Increased infrastructure costs
- Potential out-of-memory errors in long-running services

## Reproduction

I've created a minimal Docker-based reproduction that demonstrates the issue:
**Repository:** https://github.com/armoucar-neon/dspy-memory-leak-repro

```bash
# Clone and test
git clone https://github.com/armoucar-neon/dspy-memory-leak-repro
cd dspy-memory-leak-repro
docker build -t dspy-memory-leak .
docker run --rm -e OPENAI_API_KEY='your-api-key-here' dspy-memory-leak
```

## Expected Behavior

Memory should stabilize after initial module loading, with instances being properly garbage collected.

## Actual Behavior

Memory continuously grows and is never released. The test demonstrates consistent memory growth over time with repeated use of the ChainOfThought module.

## Environment

- Python: 3.11
- DSPy: 2.6.27
- OS: Linux (Docker python:3.11-bullseye)

## Additional Notes

This test specifically targets the `ChainOfThought` module. Other DSPy modules have not been tested yet and may exhibit similar behavior.

The reproduction is designed to run indefinitely and provides detailed logging to `/app/memory_growth.log` for analysis.


Discussion:
We have a FastAPI application here at the company I work for. And recently I migrated some textual prompts we used with llama-index to DSPy. Below is a pane that tracks memory in our Grafana dashboard (Last 6 hours)

<img width="939" height="378" alt="Image" src="https://github.com/user-attachments/assets/fe7c05dd-8dfb-4c3e-9cef-fc4adbc5688f" />
I've expanded the testing to include `dspy.Predict` and confirmed the memory leak affects this fundamental module as well. This suggests the issue is systemic in the DSPy framework, not limited to ChainOfThought. Since Predict is the base module used throughout DSPy, this likely impacts all applications making repeated DSPy calls.

The reproduction repository now supports testing multiple modules. To test the Predict module specifically:

```bash
docker run --rm -e OPENAI_API_KEY='your-key-here' -e DSPY_MODULE=predict dspy-memory-leak
```

Both modules show the same continuous memory growth pattern with no memory being released despite explicit cleanup and garbage collection.
I want to acknowledge this might be misuse on my side. Am I doing something wrong in how I'm creating/using/disposing of DSPy modules? Note that in my production application I don't explicitly `del` references or call `gc.collect()` - I'm only doing this in the reproduction to strengthen the case that memory cannot be cleaned up even with aggressive cleanup. Any feedback on the proper usage pattern would be appreciated.
That's crazy, thanks for opening the issue. I don't understand how a module like CoT could do this.

Is it the module history?
I've expanded my testing to include both `dspy.Predict` and `dspy.ChainOfThought` modules, and confirmed that both exhibit the same memory leak behavior, so this isn't specific to ChainOfThought. Following your suggestion about module history, I tested clearing `module.history` after each iteration, but unfortunately this didn't resolve the issue - memory continued to grow steadily despite the history being cleared and aggressive garbage collection.

I also tried replacing the global `dspy.configure()` approach with localized `dspy.context(lm=dspy.LM(...))` context managers to see if persistent global state was the culprit, but the memory leak persisted even with fresh LM instances created per iteration. My analysis suggests this is a native memory leak rather than a Python object accumulation issue, as the memory growth far exceeds what I see from Python's memory tracking tools. The leak appears to be deeper in DSPy's dependencies and happens regardless of how aggressively I clean up the Python-level objects.
@armoucar-neon  We have a global history instance in addition to module history: https://github.com/stanfordnlp/dspy/blob/6259dc46e9c07fe79e2bc7f21506e8a8105f4634/dspy/clients/base_lm.py#L9, but it does have a cap so it shouldn't trigger infinite growth. 

I am actively examining the issue, and my guess is `dspy.settings` is growing. Could you help check the `dspy.settings` size when running the docker job? Just in case this is machine-dependent. thanks!
@armoucar-neon  I found out the root cause - as suspected, `dspy.settings.trace` keeps growing. Adding a line `dspy.settings.trace = []` in the `test_iteration` function resolves the memory leak issue. An alternative is:

```
with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", cache=True), trace=[]):
````

I will file a fix for it. 
Thanks Chen! I confirmed that the previous fix for `GLOBAL_HISTORY` works ok, it does not grow further once it reaches the maximum size.



Patch:
diff --git a/dspy/clients/base_lm.py b/dspy/clients/base_lm.py
index 5247dda718..c0550b28ea 100644
--- a/dspy/clients/base_lm.py
+++ b/dspy/clients/base_lm.py
@@ -84,11 +84,9 @@ def _process_lm_response(self, response, prompt, messages, **kwargs):
             "response_model": response.model,
             "model_type": self.model_type,
         }
-        self.history.append(entry)
-        self.update_global_history(entry)
-        caller_modules = settings.caller_modules or []
-        for module in caller_modules:
-            module.history.append(entry)
+
+        self.update_history(entry)
+
         return outputs
 
     @with_callbacks
@@ -139,15 +137,32 @@ def copy(self, **kwargs):
     def inspect_history(self, n: int = 1):
         return pretty_print_history(self.history, n)
 
-    def update_global_history(self, entry):
+    def update_history(self, entry):
         if settings.disable_history:
             return
 
+        # Global LM history
         if len(GLOBAL_HISTORY) >= MAX_HISTORY_SIZE:
             GLOBAL_HISTORY.pop(0)
 
         GLOBAL_HISTORY.append(entry)
 
+        if settings.max_history_size == 0:
+            return
+
+        # dspy.LM.history
+        if len(self.history) >= settings.max_history_size:
+            self.history.pop(0)
+
+        self.history.append(entry)
+
+        # Per-module history
+        caller_modules = settings.caller_modules or []
+        for module in caller_modules:
+            if len(module.history) >= settings.max_history_size:
+                module.history.pop(0)
+            module.history.append(entry)
+
 
 def inspect_history(n: int = 1):
     """The global history shared across all LMs."""
diff --git a/dspy/dsp/utils/settings.py b/dspy/dsp/utils/settings.py
index eea41bcbe3..f5319a9f9f 100644
--- a/dspy/dsp/utils/settings.py
+++ b/dspy/dsp/utils/settings.py
@@ -26,6 +26,8 @@
     max_errors=10,  # Maximum errors before halting operations.
     # If true, async tools can be called in sync mode by getting converted to sync.
     allow_tool_async_sync_conversion=False,
+    max_history_size=10000,
+    max_trace_size=10000,
 )
 
 # Global base configuration and owner tracking
diff --git a/dspy/predict/predict.py b/dspy/predict/predict.py
index 59939b1efa..a893f0d66c 100644
--- a/dspy/predict/predict.py
+++ b/dspy/predict/predict.py
@@ -147,8 +147,10 @@ def _forward_preprocess(self, **kwargs):
 
     def _forward_postprocess(self, completions, signature, **kwargs):
         pred = Prediction.from_completions(completions, signature=signature)
-        if kwargs.pop("_trace", True) and settings.trace is not None:
+        if kwargs.pop("_trace", True) and settings.trace is not None and settings.max_trace_size > 0:
             trace = settings.trace
+            if len(trace) >= settings.max_trace_size:
+                trace.pop(0)
             trace.append((self, {**kwargs}, pred))
         return pred
 
diff --git a/dspy/utils/dummies.py b/dspy/utils/dummies.py
index 631d618c3b..6e83d11266 100644
--- a/dspy/utils/dummies.py
+++ b/dspy/utils/dummies.py
@@ -124,8 +124,7 @@ def format_answer_fields(field_names_and_values: dict[str, Any]):
             entry = {"prompt": prompt, "messages": messages, "kwargs": kwargs}
             entry = {**entry, "outputs": outputs, "usage": 0}
             entry = {**entry, "cost": 0}
-            self.history.append(entry)
-            self.update_global_history(entry)
+            self.update_history(entry)
 
         return outputs
 


Test patch:
diff --git a/tests/clients/test_lm.py b/tests/clients/test_lm.py
index b63406ba11..8aa4c89dc9 100644
--- a/tests/clients/test_lm.py
+++ b/tests/clients/test_lm.py
@@ -374,3 +374,39 @@ async def test_async_lm_call_with_cache(tmp_path):
         assert mock_alitellm_completion.call_count == 2
 
     dspy.cache = original_cache
+
+
+def test_lm_history_size_limit():
+    lm = dspy.LM(model="openai/gpt-4o-mini")
+    with dspy.context(max_history_size=5):
+        with mock.patch("litellm.completion") as mock_completion:
+            mock_completion.return_value = ModelResponse(
+                choices=[Choices(message=Message(content="test answer"))],
+                model="openai/gpt-4o-mini",
+            )
+
+            for _ in range(10):
+                lm("query")
+
+    assert len(lm.history) == 5
+
+
+def test_disable_history():
+    lm = dspy.LM(model="openai/gpt-4o-mini")
+    with dspy.context(disable_history=True):
+        with mock.patch("litellm.completion") as mock_completion:
+            mock_completion.return_value = ModelResponse(
+                choices=[Choices(message=Message(content="test answer"))],
+                model="openai/gpt-4o-mini",
+            )
+            for _ in range(10):
+                lm("query")
+
+    assert len(lm.history) == 0
+
+    with dspy.context(disable_history=False):
+        with mock.patch("litellm.completion") as mock_completion:
+            mock_completion.return_value = ModelResponse(
+                choices=[Choices(message=Message(content="test answer"))],
+                model="openai/gpt-4o-mini",
+            )
diff --git a/tests/predict/test_predict.py b/tests/predict/test_predict.py
index 37b9c87fcf..31d80fc525 100644
--- a/tests/predict/test_predict.py
+++ b/tests/predict/test_predict.py
@@ -716,10 +716,7 @@ class TestSignature(dspy.Signature):
     assert reloaded == serialized
 
     predictor = Predict(TestSignature)
-    demo = {
-        "website_info": website_info,
-        "summary": "This is a test website."
-    }
+    demo = {"website_info": website_info, "summary": "This is a test website."}
     predictor.demos = [demo]
 
     state = predictor.dump_state()
@@ -729,3 +726,41 @@ class TestSignature(dspy.Signature):
     demo_data = reloaded_state["demos"][0]
     assert demo_data["website_info"]["url"] == "https://www.example.com/"
     assert demo_data["website_info"]["created_at"] == "2021-01-01T12:00:00"
+
+
+def test_trace_size_limit():
+    program = Predict("question -> answer")
+    dspy.settings.configure(lm=DummyLM([{"answer": "Paris"}]), max_trace_size=3)
+
+    for _ in range(10):
+        program(question="What is the capital of France?")
+
+    assert len(dspy.settings.trace) == 3
+
+
+def test_disable_trace():
+    program = Predict("question -> answer")
+    dspy.settings.configure(lm=DummyLM([{"answer": "Paris"}]), trace=None)
+
+    for _ in range(10):
+        program(question="What is the capital of France?")
+
+    assert dspy.settings.trace is None
+
+
+def test_per_module_history_size_limit():
+    program = Predict("question -> answer")
+    dspy.settings.configure(lm=DummyLM([{"answer": "Paris"}]), max_history_size=5)
+
+    for _ in range(10):
+        program(question="What is the capital of France?")
+    assert len(program.history) == 5
+
+
+def test_per_module_history_disabled():
+    program = Predict("question -> answer")
+    dspy.settings.configure(lm=DummyLM([{"answer": "Paris"}]), disable_history=True)
+
+    for _ in range(10):
+        program(question="What is the capital of France?")
+    assert len(program.history) == 0