Repository: pandas-dev/pandas

Issue:
read_csv using C engine and chunksize can grow memory usage exponentially in 0.24.0rc1
#### Code Sample

```python
import pandas as pd

NUM_ROWS = 1000
CHUNKSIZE = 20

with open('test.csv', 'w') as f:
    for i in range(NUM_ROWS):
        f.write('{}\n'.format(i))
        
for chunk_index, chunk in enumerate(pd.read_csv('test.csv', chunksize=CHUNKSIZE, engine='c')):
    print(chunk_index)
```
#### Problem description

In v0.24.0rc1, using `chunksize` in `pandas.read_csv` with the C engine causes exponential memory growth (`engine='python'` works fine). 

The code sample I listed uses a very small chunksize to better illustrate the issue but the issue happens with more realistic values like `NUM_ROWS = 1000000` and `CHUNKSIZE = 1024`. The `low_memory` parameter in `pd.read_csv()` doesn't affect the behavior.

On Windows, the process becomes very slow as memory usage grows. 

On Linux, an out-of-memory exception is raised after some chunks are processed and the buffer length grows too much. For example:

<details>

```
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
```
```pytb
Traceback (most recent call last):
  File "test_csv.py", line 10, in <module>
    for chunk_index, chunk in enumerate(pd.read_csv('test.csv', chunksize=CHUNKSIZE, engine='c')):
  File "/home/meira/.conda/envs/pandas024/lib/python3.6/site-packages/pandas/io/parsers.py", line 1110, in __next__
    return self.get_chunk()
  File "/home/meira/.conda/envs/pandas024/lib/python3.6/site-packages/pandas/io/parsers.py", line 1168, in get_chunk
    return self.read(nrows=size)
  File "/home/meira/.conda/envs/pandas024/lib/python3.6/site-packages/pandas/io/parsers.py", line 1134, in read
    ret = self._engine.read(nrows)
  File "/home/meira/.conda/envs/pandas024/lib/python3.6/site-packages/pandas/io/parsers.py", line 1977, in read
    data = self._reader.read(nrows)
  File "pandas/_libs/parsers.pyx", line 893, in pandas._libs.parsers.TextReader.read
  File "pandas/_libs/parsers.pyx", line 920, in pandas._libs.parsers.TextReader._read_low_memory
  File "pandas/_libs/parsers.pyx", line 962, in pandas._libs.parsers.TextReader._read_rows
  File "pandas/_libs/parsers.pyx", line 949, in pandas._libs.parsers.TextReader._tokenize_rows
  File "pandas/_libs/parsers.pyx", line 2166, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: out of memory
```

</details>

I tried to debug the C code from the tokenizer as of 0bd454cdc9307d3a7e73403c49cc8350965628ce. The unexpected behavior seems present since 011b79fbf73b45313b47c08b4be1fc07dcb99365 which introduces these lines (and other changes) to fix #23509:

https://github.com/pandas-dev/pandas/blob/0bd454cdc9307d3a7e73403c49cc8350965628ce/pandas/_libs/src/parser/tokenizer.c#L294-L306

I'm not familiar with the code, so I could be misinterpreting it, but I believe that code block, coupled with how `self->words_cap` and `self->max_words_cap` are handled could be the source of the issue. There are some potentially misleading variables names like `nbytes` that seem to refer to the number of bytes that are later interpreted as `nbytes tokens` -- I couldn't follow what's happening but hopefully this report helps.

It seems the issue could also be related to https://github.com/pandas-dev/pandas/issues/16537 and https://github.com/pandas-dev/pandas/issues/21516 but the specific changes that cause it are newer, not present in previous releases.

#### Expected Output

```
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
```

#### Output of ``pd.show_versions()``

<details>

```
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.0.final.0
python-bits: 64
OS: Linux
OS-release: 4.12.14-lp150.12.25-default
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.0rc1
pytest: None
pip: 18.1
setuptools: 40.6.3
Cython: None
numpy: 1.15.4
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.5
pytz: 2018.9
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None
```

</details>



Discussion:
cc @gfyoung 
@PMeira : Thanks for reporting this!  I can confirm this error as well.

Did your code example previously work with `0.23.4` by any chance?
Code from the OP works for me on 0.23.4.
@gfyoung Yes, it worked fine with version `0.23.4`, just checked to be sure. 

I first noticed it from a failing test in another module that I'm trying to port to recent versions of Pandas.
@PMeira @h-vetinari : Thanks for looking into this!  Sounds like we have a regression on our hands...
@gfyoung do you have time to do this for 0.24.0? I don't think we have a release date set yet, but sometime in the next week or so?
@TomAugspurger : Yep, I'm going to look into this on the weekend.
@PMeira : Your observations are validated by what happens behind the scenes, as your numbers produce a snowball effect that causes the memory allocation to double with every iteration of reading.  

It is indeed an edge case, as your numbers work just perfectly to cause memory allocated to be powers of 2.  In fact, your "smaller example" fails for me for that reason on my local machine.

I think I have a patch for this that prevents the memory usage from growing exponentially, but I need to test to make sure I didn't break anything else with it.

Resolving patch:
diff --git a/asv_bench/benchmarks/io/csv.py b/asv_bench/benchmarks/io/csv.py
--- a/asv_bench/benchmarks/io/csv.py
+++ b/asv_bench/benchmarks/io/csv.py
@@ -214,4 +214,23 @@ def time_baseline(self):
                  names=list(string.digits[:9]))
 
 
+class ReadCSVMemoryGrowth(BaseIO):
+
+    chunksize = 20
+    num_rows = 1000
+    fname = "__test__.csv"
+
+    def setup(self):
+        with open(self.fname, "w") as f:
+            for i in range(self.num_rows):
+                f.write("{i}\n".format(i=i))
+
+    def mem_parser_chunks(self):
+        # see gh-24805.
+        result = read_csv(self.fname, chunksize=self.chunksize)
+
+        for _ in result:
+            pass
+
+
 from ..pandas_vb_common import setup  # noqa: F401


Test patch:
diff --git a/pandas/tests/io/parser/test_common.py b/pandas/tests/io/parser/test_common.py
--- a/pandas/tests/io/parser/test_common.py
+++ b/pandas/tests/io/parser/test_common.py
@@ -1916,6 +1916,24 @@ def test_filename_with_special_chars(all_parsers):
         tm.assert_frame_equal(result, df)
 
 
+def test_read_csv_memory_growth_chunksize(all_parsers):
+    # see gh-24805
+    #
+    # Let's just make sure that we don't crash
+    # as we iteratively process all chunks.
+    parser = all_parsers
+
+    with tm.ensure_clean() as path:
+        with open(path, "w") as f:
+            for i in range(1000):
+                f.write(str(i) + "\n")
+
+        result = parser.read_csv(path, chunksize=20)
+
+        for _ in result:
+            pass
+
+
 def test_read_table_deprecated(all_parsers):
     # see gh-21948
     parser = all_parsers