Repository: pandas-dev/pandas

Issue:
Memory leak in `df.to_json`
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd                                                                                                                              
import numpy as np


df = pd.DataFrame(np.random.randint(0,100,size=(100, 4)), columns=list('ABCD'))

while True:
    body = df.T.to_json()
    print("HI")


```
#### Problem description
If we repeatedly call `to_json()` on a dataframe, memory usage grows continuously:

![image](https://user-images.githubusercontent.com/880217/51640609-efe2fd80-1f31-11e9-9466-7ae2fd28f7ed.png)

#### Expected Output
I would expect memory usage to stay constant

#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]
/usr/local/lib/python3.6/dist-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.8.final.0
python-bits: 64
OS: Linux
OS-release: 4.15.0-43-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: C.UTF-8
LANG: C.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.4
pytest: None
pip: 18.1
setuptools: 40.6.3
Cython: 0.25.2
numpy: 1.15.3
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 7.1.1
sphinx: None
patsy: 0.5.1
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 3.0.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 1.0.1
sqlalchemy: None
pymysql: None
psycopg2: 2.7.5 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>



Discussion:
Source is at https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/src/ujson/python/objToJSON.c is you're interested in debugging further.
FWIW this seems to take a ton of iterations and doesn't really leak much memory, but investigations are welcome.
It's also worth isolating the to_json part from the `df.T` part.

On Thu, Jan 24, 2019 at 4:03 PM chris-b1 <notifications@github.com> wrote:

> FWIW this seems to take a ton of iterations and doesn't really leak much
> memory, but investigations are welcome.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/24889#issuecomment-457375046>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIudiRAcurMKPP8cCqhjRO7t3FAzxks5vGi26gaJpZM4aPu6O>
> .
>

@TomAugspurger sorry, that was left over from some other original code.  The leak happens with or without the transpose.
Does it also happen with other dtypes? How about an empty DataFrame?
Memory use is stable for an empty dataframe
In simple cases, there's an easy workaround.  You can just use `df.to_dict()` and pass it to python's `json.dump`.  You'll have to make some manual changes, such as manually serializing pandas timestamps.
Looked at this in more detail as I was reviewing the JSON integration. As mentioned above this issue does not affect empty dataframes, nor does it affect non-numeric frames. 

This leaks:

```python
pd.DataFrame([[1]]).to_json()
```

But this doesn't

```python
pd.DataFrame([['a']])
```

I believe I've isolated the issue to this line of code:

https://github.com/pandas-dev/pandas/blob/65466f04025474c3ee1bd2f4e49a1c5e24b76f7b/pandas/_libs/src/ujson/python/objToJSON.c#L716

Removing that prevented the example from the OP from leaking. Unfortunately it caused segfaults in the rest of the code base so have to debug that further, but will push a PR if I can figure it out.

@jbrockmendel you might have some thoughts here as well. You called this condition out as looking wonky before and it does seem to cause the leak here, just need to figure out what the actual intent is

Resolving patch:
diff --git a/asv_bench/benchmarks/io/json.py b/asv_bench/benchmarks/io/json.py
--- a/asv_bench/benchmarks/io/json.py
+++ b/asv_bench/benchmarks/io/json.py
@@ -124,4 +124,26 @@ def time_float_int_str_lines(self, orient):
         self.df_int_float_str.to_json(self.fname, orient='records', lines=True)
 
 
+class ToJSONMem:
+
+    def setup_cache(self):
+        df = DataFrame([[1]])
+        frames = {
+            'int': df,
+            'float': df.astype(float),
+        }
+
+        return frames
+
+    def peakmem_int(self, frames):
+        df = frames['int']
+        for _ in range(100_000):
+            df.to_json()
+
+    def peakmem_float(self, frames):
+        df = frames['float']
+        for _ in range(100_000):
+            df.to_json()
+
+
 from ..pandas_vb_common import setup  # noqa: F401


Test patch: