Repository: sympy/sympy
Issue category: Bug Report

Issue:
Memory Leak in `sympy.lambdify`
Hi there,

I'm working with an [algorithm](https://github.com/SymposiumOrganization/NeuralSymbolicRegressionThatScales) that relies on calling `sympy.lambdify` hundreds of millions of times (~200M) and noticed the memory usage of the process steadily creeping up and eventually crashing the program.

I was able to trace the problem to the following (see **Demo** below).

Notably, the memory usage still increases despite
1. lambdifying the same equation again
2. "deleting" the lambdified equation

**System Information:**
WSL2 (Windows 10.0.22631 Build 22631)

```
> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy
```
```
> python --version
Python 3.11.10
```
```
> pip freeze
mpmath==1.3.0
psutil==6.1.0
sympy==1.13.3
```


### Demo:


```python
# sympy_memory.py
import argparse
import psutil

import sympy


if __name__ == '__main__':
    # Parse command line arguments
    parser = argparse.ArgumentParser(description='Run equation n times')
    parser.add_argument('-n', type=int, default=1, help='Number of repetitions')
    args = parser.parse_args()

    # Get the current process to monitor memory usage
    process = psutil.Process()

    # Parse an example equation
    equation_str = 'x'
    parsed_equation = sympy.sympify(equation_str)

    print("Memory Usage:")

    for i in range(args.n):
        # Lambdifying the (same) equation in each iteration
        lambdified_equation = sympy.lambdify('x', parsed_equation)

        # Print memory usage every 10000 iterations
        if i % 10000 == 0:
            mem_usage = process.memory_info().rss / 1024 / 1024  # Convert to MB
            print(f"{i}/{args.n}: {mem_usage:.2f} MB")

        # "Deleting" the lambdified equation
        del lambdified_equation
```

```
> python sympy_memory.py -n 100000
Memory Usage:
0/100000: 49.91 MB
10000/100000: 54.29 MB
20000/100000: 59.27 MB
30000/100000: 64.17 MB
40000/100000: 68.04 MB
50000/100000: 72.05 MB
60000/100000: 75.91 MB
70000/100000: 79.78 MB
80000/100000: 83.45 MB
90000/100000: 89.08 MB
```

**Observation:** The memory usage increases by about ~~+390KB~~ (correction: 390B https://github.com/sympy/sympy/issues/27216#issuecomment-2453168997) per lambdified equation, totalling +39MB over 100k lambdifications and deletions.

### Control (to exclude external factors):

```diff
    parsed_equation = sympy.sympify(equation_str)
+    lambdified_equation = sympy.lambdify('x', parsed_equation)

    print("Memory Usage:")

    for i in range(args.n):
-        # Lambdifying the (same) equation in each iteration
-        lambdified_equation = sympy.lambdify('x', parsed_equation)

        # Print memory usage every 10000 iterations
        if i % 10000 == 0:
            mem_usage = process.memory_info().rss / 1024 / 1024  # Convert to MB
            print(f"{i}/{args.n}: {mem_usage:.2f} MB")

-        # "Deleting" the lambdified equation
-        del lambdified_equation
```

```
> python sympy_memory.py -n 100000
Memory Usage:
0/100000: 49.91 MB
10000/100000: 49.91 MB
20000/100000: 49.91 MB
30000/100000: 49.91 MB
40000/100000: 49.91 MB
50000/100000: 49.91 MB
60000/100000: 49.91 MB
70000/100000: 49.91 MB
80000/100000: 49.91 MB
90000/100000: 49.91 MB
```

**Observation:** The memory usage does not increase.
Memory Leak in `sympy.lambdify`
Hi there,

I'm working with an [algorithm](https://github.com/SymposiumOrganization/NeuralSymbolicRegressionThatScales) that relies on calling `sympy.lambdify` hundreds of millions of times (~200M) and noticed the memory usage of the process steadily creeping up and eventually crashing the program.

I was able to trace the problem to the following (see **Demo** below).

Notably, the memory usage still increases despite
1. lambdifying the same equation again
2. "deleting" the lambdified equation

**System Information:**
WSL2 (Windows 10.0.22631 Build 22631)

```
> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy
```
```
> python --version
Python 3.11.10
```
```
> pip freeze
mpmath==1.3.0
psutil==6.1.0
sympy==1.13.3
```


### Demo:


```python
# sympy_memory.py
import argparse
import psutil

import sympy


if __name__ == '__main__':
    # Parse command line arguments
    parser = argparse.ArgumentParser(description='Run equation n times')
    parser.add_argument('-n', type=int, default=1, help='Number of repetitions')
    args = parser.parse_args()

    # Get the current process to monitor memory usage
    process = psutil.Process()

    # Parse an example equation
    equation_str = 'x'
    parsed_equation = sympy.sympify(equation_str)

    print("Memory Usage:")

    for i in range(args.n):
        # Lambdifying the (same) equation in each iteration
        lambdified_equation = sympy.lambdify('x', parsed_equation)

        # Print memory usage every 10000 iterations
        if i % 10000 == 0:
            mem_usage = process.memory_info().rss / 1024 / 1024  # Convert to MB
            print(f"{i}/{args.n}: {mem_usage:.2f} MB")

        # "Deleting" the lambdified equation
        del lambdified_equation
```

```
> python sympy_memory.py -n 100000
Memory Usage:
0/100000: 49.91 MB
10000/100000: 54.29 MB
20000/100000: 59.27 MB
30000/100000: 64.17 MB
40000/100000: 68.04 MB
50000/100000: 72.05 MB
60000/100000: 75.91 MB
70000/100000: 79.78 MB
80000/100000: 83.45 MB
90000/100000: 89.08 MB
```

**Observation:** The memory usage increases by about ~~+390KB~~ (correction: 390B https://github.com/sympy/sympy/issues/27216#issuecomment-2453168997) per lambdified equation, totalling +39MB over 100k lambdifications and deletions.

### Control (to exclude external factors):

```diff
    parsed_equation = sympy.sympify(equation_str)
+    lambdified_equation = sympy.lambdify('x', parsed_equation)

    print("Memory Usage:")

    for i in range(args.n):
-        # Lambdifying the (same) equation in each iteration
-        lambdified_equation = sympy.lambdify('x', parsed_equation)

        # Print memory usage every 10000 iterations
        if i % 10000 == 0:
            mem_usage = process.memory_info().rss / 1024 / 1024  # Convert to MB
            print(f"{i}/{args.n}: {mem_usage:.2f} MB")

-        # "Deleting" the lambdified equation
-        del lambdified_equation
```

```
> python sympy_memory.py -n 100000
Memory Usage:
0/100000: 49.91 MB
10000/100000: 49.91 MB
20000/100000: 49.91 MB
30000/100000: 49.91 MB
40000/100000: 49.91 MB
50000/100000: 49.91 MB
60000/100000: 49.91 MB
70000/100000: 49.91 MB
80000/100000: 49.91 MB
90000/100000: 49.91 MB
```

**Observation:** The memory usage does not increase.


Hints:
> The memory usage increases by about +390KB per lambdified equation

I assume you mean per 10000 lambdified equations so it is about 400 bytes per lambdified equation.
My guess is that each call to lambdify creates a Dummy and then something creates a polynomial ring with that dummy and the polynomial ring never gets deleted.
The memory leak is from this line:
https://github.com/sympy/sympy/blob/d293133e81194adc11177729af91c970f092a6e7/sympy/utilities/lambdify.py#L908
I assume that the purpose of the line is something to do with tracebacks but I don't know lambdify so well.
> > The memory usage increases by about +390KB per lambdified equation
> 
> I assume you mean per 10000 lambdified equations so it is about 400 bytes per lambdified equation.

Correct, (89.08MB - 49.91MB) / 100,000 is about 390B per equation.
I experimented with wrapping the function in an object that would clear the cache upon deletion of the function which significantly reduces the accumulated memory.

### Current:
https://github.com/sympy/sympy/blob/d293133e81194adc11177729af91c970f092a6e7/sympy/utilities/lambdify.py#L933

### Experimental:
Modified `sympy/utilities/lambdify.py`:
``` diff
    func.__doc__ = (
        "Created with lambdify. Signature:\n\n"
        "{sig}\n\n"
        "Expression:\n\n"
        "{expr}\n\n"
        "Source code:\n\n"
        "{src}\n\n"
        "Imported modules:\n\n"
        "{imp_mods}"
        ).format(sig=sig, expr=expr_str, src=src_str, imp_mods='\n'.join(imp_mod_lines))

-    return func
+    return DeletableLambdaFunction(func)
+
+class DeletableLambdaFunction:
+    def __init__(self, func):
+        self.func = func
+
+    def __call__(self, *args, **kwargs):
+        return self.func(*args, **kwargs)
+    
+    def __del__(self):
+        linecache.clearcache()
+        del self.func
+
+    def __doc__(self):
+        return self.func.__doc__

def _module_present(modname, modlist):
    if modname in modlist:
        return True
    for m in modlist:
        if hasattr(m, '__name__') and m.__name__ == modname:
            return True
    return False
```

```
> python sympy_memory.py -n 100000
Memory Usage:
0/100000: 49.80 MB
10000/100000: 50.06 MB
20000/100000: 51.12 MB
30000/100000: 51.12 MB
40000/100000: 51.12 MB
50000/100000: 51.12 MB
60000/100000: 51.38 MB
70000/100000: 51.38 MB
80000/100000: 51.46 MB
90000/100000: 51.46 MB
```
First question is whether there is a good reason to add to linecache at all.
It was added in 8a892b2 from gh-14739
Inserting into `linecache` is needed for the debugger to be able to show the source of the compiled function. Also clearing linecache is a global operation which interfers with other libraries, so we would need to selectively delete only the relevant entries from a `__del__` method.

edit: There's a [relevant issue](https://github.com/pytorch/pytorch/issues/62917) at pytorch which links to their current implementation and their plan to move to lazycache. Perhaps that's the proper fix, I don't know for sure.
A simple solution would be an option for `lambdify` to disable adding to linecache like `lambdify(..., linecache=False)`.
I don't see how lazycache would solve this issue. lazycache lets you delay computation of the lines (the idea is to avoid unnecessary I/O in linecache). But for lambdify that would either mean just storing the lines elsewhere in the function closure, or recomputing lambdify in it. The latter could be bug prone if any part of lambdify relied on some state that changed since the original call. 

I think it should be possible to fix this by using `weakref.finalize` https://docs.python.org/3/library/weakref.html#weakref.finalize to clear the linecache entry whenever the function is garbage collected. (it would also be nice if linecache supported this directly; maybe someone can open an issue/PR to CPython)
> The memory usage increases by about +390KB per lambdified equation

I assume you mean per 10000 lambdified equations so it is about 400 bytes per lambdified equation.
My guess is that each call to lambdify creates a Dummy and then something creates a polynomial ring with that dummy and the polynomial ring never gets deleted.
The memory leak is from this line:
https://github.com/sympy/sympy/blob/d293133e81194adc11177729af91c970f092a6e7/sympy/utilities/lambdify.py#L908
I assume that the purpose of the line is something to do with tracebacks but I don't know lambdify so well.
> > The memory usage increases by about +390KB per lambdified equation
> 
> I assume you mean per 10000 lambdified equations so it is about 400 bytes per lambdified equation.

Correct, (89.08MB - 49.91MB) / 100,000 is about 390B per equation.
I experimented with wrapping the function in an object that would clear the cache upon deletion of the function which significantly reduces the accumulated memory.

### Current:
https://github.com/sympy/sympy/blob/d293133e81194adc11177729af91c970f092a6e7/sympy/utilities/lambdify.py#L933

### Experimental:
Modified `sympy/utilities/lambdify.py`:
``` diff
    func.__doc__ = (
        "Created with lambdify. Signature:\n\n"
        "{sig}\n\n"
        "Expression:\n\n"
        "{expr}\n\n"
        "Source code:\n\n"
        "{src}\n\n"
        "Imported modules:\n\n"
        "{imp_mods}"
        ).format(sig=sig, expr=expr_str, src=src_str, imp_mods='\n'.join(imp_mod_lines))

-    return func
+    return DeletableLambdaFunction(func)
+
+class DeletableLambdaFunction:
+    def __init__(self, func):
+        self.func = func
+
+    def __call__(self, *args, **kwargs):
+        return self.func(*args, **kwargs)
+    
+    def __del__(self):
+        linecache.clearcache()
+        del self.func
+
+    def __doc__(self):
+        return self.func.__doc__

def _module_present(modname, modlist):
    if modname in modlist:
        return True
    for m in modlist:
        if hasattr(m, '__name__') and m.__name__ == modname:
            return True
    return False
```

```
> python sympy_memory.py -n 100000
Memory Usage:
0/100000: 49.80 MB
10000/100000: 50.06 MB
20000/100000: 51.12 MB
30000/100000: 51.12 MB
40000/100000: 51.12 MB
50000/100000: 51.12 MB
60000/100000: 51.38 MB
70000/100000: 51.38 MB
80000/100000: 51.46 MB
90000/100000: 51.46 MB
```
First question is whether there is a good reason to add to linecache at all.
It was added in 8a892b2 from gh-14739
Inserting into `linecache` is needed for the debugger to be able to show the source of the compiled function. Also clearing linecache is a global operation which interfers with other libraries, so we would need to selectively delete only the relevant entries from a `__del__` method.

edit: There's a [relevant issue](https://github.com/pytorch/pytorch/issues/62917) at pytorch which links to their current implementation and their plan to move to lazycache. Perhaps that's the proper fix, I don't know for sure.
A simple solution would be an option for `lambdify` to disable adding to linecache like `lambdify(..., linecache=False)`.
I don't see how lazycache would solve this issue. lazycache lets you delay computation of the lines (the idea is to avoid unnecessary I/O in linecache). But for lambdify that would either mean just storing the lines elsewhere in the function closure, or recomputing lambdify in it. The latter could be bug prone if any part of lambdify relied on some state that changed since the original call. 

I think it should be possible to fix this by using `weakref.finalize` https://docs.python.org/3/library/weakref.html#weakref.finalize to clear the linecache entry whenever the function is garbage collected. (it would also be nice if linecache supported this directly; maybe someone can open an issue/PR to CPython)

Patch:
diff --git a/sympy/utilities/lambdify.py b/sympy/utilities/lambdify.py
index a84d1a1c26c1..518f5cb67bf5 100644
--- a/sympy/utilities/lambdify.py
+++ b/sympy/utilities/lambdify.py
@@ -11,6 +11,7 @@
 import keyword
 import textwrap
 import linecache
+import weakref
 
 # Required despite static analysis claiming it is not used
 from sympy.external import import_module # noqa:F401
@@ -907,8 +908,17 @@ def _lambdifygenerated(x):
     # mtime has to be None or else linecache.checkcache will remove it
     linecache.cache[filename] = (len(funcstr), None, funcstr.splitlines(True), filename) # type: ignore
 
+    # Remove the entry from the linecache when the object is garbage collected
+    def cleanup_linecache(filename):
+        def _cleanup():
+            if filename in linecache.cache:
+                del linecache.cache[filename]
+        return _cleanup
+
     func = funclocals[funcname]
 
+    weakref.finalize(func, cleanup_linecache(filename))
+
     # Apply the docstring
     sig = "func({})".format(", ".join(str(i) for i in names))
     sig = textwrap.fill(sig, subsequent_indent=' '*8)


Test patch:
diff --git a/sympy/utilities/tests/test_lambdify.py b/sympy/utilities/tests/test_lambdify.py
index 4a82290569ea..428cbaed92b6 100644
--- a/sympy/utilities/tests/test_lambdify.py
+++ b/sympy/utilities/tests/test_lambdify.py
@@ -1,6 +1,8 @@
 from itertools import product
 import math
 import inspect
+import linecache
+import gc
 
 import mpmath
 
@@ -981,6 +983,18 @@ def test_lambdify_docstring():
     assert func.__doc__.splitlines()[:len(ref)] == ref
 
 
+def test_lambdify_linecache():
+    func = lambdify(x, x + 1)
+    source = 'def _lambdifygenerated(x):\n    return x + 1\n'
+    assert inspect.getsource(func) == source
+    filename = inspect.getsourcefile(func)
+    assert filename.startswith('<lambdifygenerated-')
+    assert filename in linecache.cache
+    assert linecache.cache[filename] == (len(source), None, source.splitlines(True), filename)
+    del func
+    gc.collect()
+    assert filename not in linecache.cache
+
 #================== Test special printers ==========================