Repository: pandas-dev/pandas

Issue:
Memory leak with .rolling().max() in pandas 0.24.2 
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd
df = pd.read_csv('file.csv', index_col=0, parse_dates=True)
while True:
    df['close'].rolling(4000).max()
```
#### Problem description

Memory leak which shuts down my application. This occurs in pandas 0.24.2 but not in pandas 0.23.4. My 16 GB memory gets filled in a few hours of running this code. file.csv is attached in zip, the memory leak might only occur on certain data.
[file.csv.zip](https://github.com/pandas-dev/pandas/files/3013836/file.csv.zip)

#### Expected Output
No memory leaks.

#### Output of ``pd.show_versions()``

<details>
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.8.final.0
python-bits: 64
OS: Linux
OS-release: 4.15.0-46-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.2
pytest: None
pip: 19.0.3
setuptools: 38.4.0
Cython: None
numpy: 1.14.5
scipy: None
pyarrow: None
xarray: None
IPython: 7.1.1
sphinx: None
patsy: None
dateutil: 2.7.5
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: 4.7.1
html5lib: None
sqlalchemy: 1.2.14
pymysql: None
psycopg2: 2.7.6.1 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

</details>



Discussion:
Can you profile and try to isolate the issue?
```python
import os
import psutil
import numpy as np
import pandas as pd

process = psutil.Process(os.getpid())
i = 0

def sink():
    global i
    i += 1
    if i % 100 == 0:
        mem = process.memory_info().rss / 1e6
        print("mem %fMB" % mem)

while True:
    pd.Series(data=np.random.rand(5000)).rolling(4000).max()
    sink()
```

I was wrong, it has nothing to do with the dataset I provided. This is a pretty huge bug in pandas imo.
Thanks for the code sample. Investigation and PRs are always welcome!
The bug does not occur with rolling().mean(), but it does with rolling().min().
It does look like the min/max implementation is the only window func with calls to malloc/free:

https://github.com/pandas-dev/pandas/blob/ac318d26cdbe3d10030f4863b84a06e938da9bba/pandas/_libs/window.pyx#L1372

Per Cython docs it may be preferable to use the C-API functions for better memory management and reporting back to the Python layer:

https://cython.readthedocs.io/en/latest/src/tutorial/memory_allocation.html#memory-allocation

Would take a PR trying that or other ideas for sure
May also be that the GIL is released when free is called in current implementation
After some testing, it doesn't seem to have anything to do with the GIL or using the C-API vs direct malloc/free calls.

It looks like this bug was introduced when separating variable/fixed into separate functions.  For some reason, passing some of the cdef variables to separate functions (likely `starti` and `endi` in this case) are what cause the memory leak.

A simple fix is to just combine the functions into one large function again, but it may be better to work out an alternative that doesn't cause the leak, like using pointers or perhaps a [memory view](https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html#memoryviews).

I can take a shot at a proper fix, but it may be a couple of days until I have a PR.
Sounds good thanks for investigating @ArtificialQualia !

Resolving patch:
diff --git a/asv_bench/benchmarks/rolling.py b/asv_bench/benchmarks/rolling.py
--- a/asv_bench/benchmarks/rolling.py
+++ b/asv_bench/benchmarks/rolling.py
@@ -113,4 +113,20 @@ def time_quantile(self, constructor, window, dtype, percentile,
         self.roll.quantile(percentile, interpolation=interpolation)
 
 
+class PeakMemFixed(object):
+
+    def setup(self):
+        N = 10
+        arr = 100 * np.random.random(N)
+        self.roll = pd.Series(arr).rolling(10)
+
+    def peakmem_fixed(self):
+        # GH 25926
+        # This is to detect memory leaks in rolling operations.
+        # To save time this is only ran on one method.
+        # 6000 iterations is enough for most types of leaks to be detected
+        for x in range(6000):
+            self.roll.max()
+
+
 from .pandas_vb_common import setup  # noqa: F401


Test patch: