Repository: sympy/sympy
Issue category: Bug Report

Memory Leak in `sympy.lambdify`
Hi there,

I'm working with an [algorithm](https://github.com/SymposiumOrganization/NeuralSymbolicRegressionThatScales) that relies on calling `sympy.lambdify` hundreds of millions of times (~200M) and noticed the memory usage of the process steadily creeping up and eventually crashing the program.

I was able to trace the problem to the following (see **Demo** below).

Notably, the memory usage still increases despite
1. lambdifying the same equation again
2. "deleting" the lambdified equation

**System Information:**
WSL2 (Windows 10.0.22631 Build 22631)

```
> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy
```
```
> python --version
Python 3.11.10
```
```
> pip freeze
mpmath==1.3.0
psutil==6.1.0
sympy==1.13.3
```


### Demo:


```python
# sympy_memory.py
import argparse
import psutil

import sympy


if __name__ == '__main__':
    # Parse command line arguments
    parser = argparse.ArgumentParser(description='Run equation n times')
    parser.add_argument('-n', type=int, default=1, help='Number of repetitions')
    args = parser.parse_args()

    # Get the current process to monitor memory usage
    process = psutil.Process()

    # Parse an example equation
    equation_str = 'x'
    parsed_equation = sympy.sympify(equation_str)

    print("Memory Usage:")

    for i in range(args.n):
        # Lambdifying the (same) equation in each iteration
        lambdified_equation = sympy.lambdify('x', parsed_equation)

        # Print memory usage every 10000 iterations
        if i % 10000 == 0:
            mem_usage = process.memory_info().rss / 1024 / 1024  # Convert to MB
            print(f"{i}/{args.n}: {mem_usage:.2f} MB")

        # "Deleting" the lambdified equation
        del lambdified_equation
```

```
> python sympy_memory.py -n 100000
Memory Usage:
0/100000: 49.91 MB
10000/100000: 54.29 MB
20000/100000: 59.27 MB
30000/100000: 64.17 MB
40000/100000: 68.04 MB
50000/100000: 72.05 MB
60000/100000: 75.91 MB
70000/100000: 79.78 MB
80000/100000: 83.45 MB
90000/100000: 89.08 MB
```

**Observation:** The memory usage increases by about ~~+390KB~~ (correction: 390B https://github.com/sympy/sympy/issues/27216#issuecomment-2453168997) per lambdified equation, totalling +39MB over 100k lambdifications and deletions.

### Control (to exclude external factors):

```diff
    parsed_equation = sympy.sympify(equation_str)
+    lambdified_equation = sympy.lambdify('x', parsed_equation)

    print("Memory Usage:")

    for i in range(args.n):
-        # Lambdifying the (same) equation in each iteration
-        lambdified_equation = sympy.lambdify('x', parsed_equation)

        # Print memory usage every 10000 iterations
        if i % 10000 == 0:
            mem_usage = process.memory_info().rss / 1024 / 1024  # Convert to MB
            print(f"{i}/{args.n}: {mem_usage:.2f} MB")

-        # "Deleting" the lambdified equation
-        del lambdified_equation
```

```
> python sympy_memory.py -n 100000
Memory Usage:
0/100000: 49.91 MB
10000/100000: 49.91 MB
20000/100000: 49.91 MB
30000/100000: 49.91 MB
40000/100000: 49.91 MB
50000/100000: 49.91 MB
60000/100000: 49.91 MB
70000/100000: 49.91 MB
80000/100000: 49.91 MB
90000/100000: 49.91 MB
```

**Observation:** The memory usage does not increase.
Memory Leak in `sympy.lambdify`
Hi there,

I'm working with an [algorithm](https://github.com/SymposiumOrganization/NeuralSymbolicRegressionThatScales) that relies on calling `sympy.lambdify` hundreds of millions of times (~200M) and noticed the memory usage of the process steadily creeping up and eventually crashing the program.

I was able to trace the problem to the following (see **Demo** below).

Notably, the memory usage still increases despite
1. lambdifying the same equation again
2. "deleting" the lambdified equation

**System Information:**
WSL2 (Windows 10.0.22631 Build 22631)

```
> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy
```
```
> python --version
Python 3.11.10
```
```
> pip freeze
mpmath==1.3.0
psutil==6.1.0
sympy==1.13.3
```


### Demo:


```python
# sympy_memory.py
import argparse
import psutil

import sympy


if __name__ == '__main__':
    # Parse command line arguments
    parser = argparse.ArgumentParser(description='Run equation n times')
    parser.add_argument('-n', type=int, default=1, help='Number of repetitions')
    args = parser.parse_args()

    # Get the current process to monitor memory usage
    process = psutil.Process()

    # Parse an example equation
    equation_str = 'x'
    parsed_equation = sympy.sympify(equation_str)

    print("Memory Usage:")

    for i in range(args.n):
        # Lambdifying the (same) equation in each iteration
        lambdified_equation = sympy.lambdify('x', parsed_equation)

        # Print memory usage every 10000 iterations
        if i % 10000 == 0:
            mem_usage = process.memory_info().rss / 1024 / 1024  # Convert to MB
            print(f"{i}/{args.n}: {mem_usage:.2f} MB")

        # "Deleting" the lambdified equation
        del lambdified_equation
```

```
> python sympy_memory.py -n 100000
Memory Usage:
0/100000: 49.91 MB
10000/100000: 54.29 MB
20000/100000: 59.27 MB
30000/100000: 64.17 MB
40000/100000: 68.04 MB
50000/100000: 72.05 MB
60000/100000: 75.91 MB
70000/100000: 79.78 MB
80000/100000: 83.45 MB
90000/100000: 89.08 MB
```

**Observation:** The memory usage increases by about ~~+390KB~~ (correction: 390B https://github.com/sympy/sympy/issues/27216#issuecomment-2453168997) per lambdified equation, totalling +39MB over 100k lambdifications and deletions.

### Control (to exclude external factors):

```diff
    parsed_equation = sympy.sympify(equation_str)
+    lambdified_equation = sympy.lambdify('x', parsed_equation)

    print("Memory Usage:")

    for i in range(args.n):
-        # Lambdifying the (same) equation in each iteration
-        lambdified_equation = sympy.lambdify('x', parsed_equation)

        # Print memory usage every 10000 iterations
        if i % 10000 == 0:
            mem_usage = process.memory_info().rss / 1024 / 1024  # Convert to MB
            print(f"{i}/{args.n}: {mem_usage:.2f} MB")

-        # "Deleting" the lambdified equation
-        del lambdified_equation
```

```
> python sympy_memory.py -n 100000
Memory Usage:
0/100000: 49.91 MB
10000/100000: 49.91 MB
20000/100000: 49.91 MB
30000/100000: 49.91 MB
40000/100000: 49.91 MB
50000/100000: 49.91 MB
60000/100000: 49.91 MB
70000/100000: 49.91 MB
80000/100000: 49.91 MB
90000/100000: 49.91 MB
```

**Observation:** The memory usage does not increase.


Functions to edit:
sympy/utilities/lambdify.py:lambdify

Patch:
diff --git a/sympy/utilities/lambdify.py b/sympy/utilities/lambdify.py
index a84d1a1c26c1..518f5cb67bf5 100644
--- a/sympy/utilities/lambdify.py
+++ b/sympy/utilities/lambdify.py
@@ -11,6 +11,7 @@
 import keyword
 import textwrap
 import linecache
+import weakref
 
 # Required despite static analysis claiming it is not used
 from sympy.external import import_module # noqa:F401
@@ -907,8 +908,17 @@ def _lambdifygenerated(x):
     # mtime has to be None or else linecache.checkcache will remove it
     linecache.cache[filename] = (len(funcstr), None, funcstr.splitlines(True), filename) # type: ignore
 
+    # Remove the entry from the linecache when the object is garbage collected
+    def cleanup_linecache(filename):
+        def _cleanup():
+            if filename in linecache.cache:
+                del linecache.cache[filename]
+        return _cleanup
+
     func = funclocals[funcname]
 
+    weakref.finalize(func, cleanup_linecache(filename))
+
     # Apply the docstring
     sig = "func({})".format(", ".join(str(i) for i in names))
     sig = textwrap.fill(sig, subsequent_indent=' '*8)

Test patch: diff --git a/sympy/utilities/tests/test_lambdify.py b/sympy/utilities/tests/test_lambdify.py
index 4a82290569ea..428cbaed92b6 100644
--- a/sympy/utilities/tests/test_lambdify.py
+++ b/sympy/utilities/tests/test_lambdify.py
@@ -1,6 +1,8 @@
 from itertools import product
 import math
 import inspect
+import linecache
+import gc
 
 import mpmath
 
@@ -981,6 +983,18 @@ def test_lambdify_docstring():
     assert func.__doc__.splitlines()[:len(ref)] == ref
 
 
+def test_lambdify_linecache():
+    func = lambdify(x, x + 1)
+    source = 'def _lambdifygenerated(x):\n    return x + 1\n'
+    assert inspect.getsource(func) == source
+    filename = inspect.getsourcefile(func)
+    assert filename.startswith('<lambdifygenerated-')
+    assert filename in linecache.cache
+    assert linecache.cache[filename] == (len(source), None, source.splitlines(True), filename)
+    del func
+    gc.collect()
+    assert filename not in linecache.cache
+
 #================== Test special printers ==========================