Repository: pandas-dev/pandas

Issue:
read_csv with iterator=True does not seem to work as expected without chunksize
Hi there,

I tested:

``` python
data = """A,B,C
foo,1,2,3
bar,4,5,6
baz,7,8,9
"""
reader = pd.read_csv(StringIO(data), iterator=True)
```

I thought that I could do:

```
for row in reader:
    print(row)
```

since `reader` is iterable. Unfortunately, it calls the generator `TextFileReader.__iter__`:

``` python
def __iter__(self):
    try:
        while True:
            yield self.read(self.chunksize)
    except StopIteration:
        pass
```

where `self.chunksize` is `None`. Maybe set `self.chunsize` to 1 when it's not
defined and there is `iterator=True`. I'll propose a patch as soon as possible
--- today or tomorrow.

Best regards,
Damien G.



Discussion:
Well, `iterator[=False]`  occurs in the kwd arguments of `read_csv` but commented in the dictionary `_parser_defaults`. I wonder if I set, "in a dummy way", `chunksize=1` in this dictionary or if it's quite more complicated.

I'm writing a test and waiting for some comments about this.

**Edit**: Aarghhh. Force `self.chunksize=1` in `TextFileReader` throws me a pretty SegFault :-) Actually, it's quite unhappy to have a new key `iterator` in `*kwds` (something which deals with `_get_options_with_default`). Quite confused.

Damien G.

Wouldn't making empty DataFrames be Falsey also solve this ?

Edit: No it wouldn't!

I was thinking you could do:

```
while (yield self.read(self.chunksize)): pass
```

but that is probably semantically different.

i believe this was fixed #3406 

@jreback This still causes an infinite loop (isn't this the issue?):

```
In [6]: for row in reader:
    print row
```

@jreback I think it's not. I'm on the latest commit of master and the

``` python
while True:
    yield self.read(self.chunksize)
```

causes the issue when `self.chunksize=None`.
@hayd Yes, the infinite loop is the issue. Sorry, I was not very clear about that.

Update.

``` python
>>> reader = pd.read_csv('SampleData.csv', iterator=True, engine='python')
>>> reader.chunksize is None
True
>>> for row in reader:
        print(row)
     A  B  C
foo  1  2  3
bar  4  5  6
baz  7  8  9
```

No infinite loop here but returns the full `DataFrame` directly. By the way, the
Python parser engine does not seem to take into account the value of
`chunksize`.

``` python
>>> reader = pd.read_csv('SampleData.csv', chunksize=1, engine='python')
>>> for row in reader:
        print(row)
     A  B  C
foo  1  2  3
bar  4  5  6
     A  B  C
baz  7  8  9
```

loop in `reader` 2 by 2 `DataFrame`. I'll open an other issue about this.

For the C engine:

``` python
>>> reader = pd.read_csv('SampleData.csv', iterator=True, engine='c)
>>> reader.chunksize is None
True
>>> for row in reader:
        print(row)
Empty DataFrame
Columns: [Region, Rep, Item, Units, Unit Cost, Total]
Index: []
Empty DataFrame
Columns: [Region, Rep, Item, Units, Unit Cost, Total]
Index: []
...
```

infinitely. I have the Segmentation Fault when I try to pass the `iterator`
parameter to `TextFileReader` as described above.

leave it all in this issue its all related

OK. The SegFault is my bad (inconsistency between my source and install dirs and `*.so` files generated by Cython).

Take a look at https://github.com/garaud/pandas/commit/b0d890308f4875657df433b1403ce995400e4d73

Works well with `TestCParserHighMemory`, `TestCParserLowMemory` but fails with `TestPythonParser`. I don't get why Python engine carries out a full loop on `TextFileReader` when `chunksize=None`. Keep going, keep going...

Damien G.

if you specify `iterator=True` but not `chunksize`, this is tantamount to reading the entire file; I don't think you can assume `chunksize=1` or really any number. If you want 1, then you need to specify it.


Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -186,7 +186,7 @@ def _read(filepath_or_buffer, kwds):
             kwds['parse_dates'] = True
 
     # Extract some of the arguments (pass chunksize on).
-    iterator = kwds.pop('iterator', False)
+    iterator = kwds.get('iterator', False)
     nrows = kwds.pop('nrows', None)
     chunksize = kwds.get('chunksize', None)
 
@@ -569,8 +569,11 @@ def _clean_options(self, options, engine):
 
     def __iter__(self):
         try:
-            while True:
-                yield self.read(self.chunksize)
+            if self.chunksize:
+                while True:
+                    yield self.read(self.chunksize)
+            else:
+                yield self.read()
         except StopIteration:
             pass
 
@@ -1594,47 +1597,58 @@ def _rows_to_cols(self, content):
     def _get_lines(self, rows=None):
         source = self.data
         lines = self.buf
+        new_rows = None
 
         # already fetched some number
         if rows is not None:
-            rows -= len(self.buf)
 
-        if isinstance(source, list):
-            if self.pos > len(source):
-                raise StopIteration
-            if rows is None:
-                lines.extend(source[self.pos:])
-                self.pos = len(source)
+            # we already have the lines in the buffer
+            if len(self.buf) >= rows:
+                new_rows, self.buf = self.buf[:rows], self.buf[rows:]
+
+            # need some lines
             else:
-                lines.extend(source[self.pos:self.pos + rows])
-                self.pos += rows
-        else:
-            new_rows = []
-            try:
-                if rows is not None:
-                    for _ in xrange(rows):
-                        new_rows.append(next(source))
-                    lines.extend(new_rows)
+                rows -= len(self.buf)
+
+        if new_rows is None:
+            if isinstance(source, list):
+                if self.pos > len(source):
+                    raise StopIteration
+                if rows is None:
+                    lines.extend(source[self.pos:])
+                    self.pos = len(source)
                 else:
-                    rows = 0
-                    while True:
-                        try:
+                    lines.extend(source[self.pos:self.pos + rows])
+                    self.pos += rows
+            else:
+                new_rows = []
+                try:
+                    if rows is not None:
+                        for _ in xrange(rows):
                             new_rows.append(next(source))
-                            rows += 1
-                        except csv.Error, inst:
-                            if 'newline inside string' in str(inst):
-                                row_num = str(self.pos + rows)
-                                msg = ('EOF inside string starting with line '
-                                       + row_num)
-                                raise Exception(msg)
-                            raise
-            except StopIteration:
-                lines.extend(new_rows)
-                if len(lines) == 0:
-                    raise
-            self.pos += len(new_rows)
+                        lines.extend(new_rows)
+                    else:
+                        rows = 0
+                        while True:
+                            try:
+                                new_rows.append(next(source))
+                                rows += 1
+                            except csv.Error, inst:
+                                if 'newline inside string' in str(inst):
+                                    row_num = str(self.pos + rows)
+                                    msg = ('EOF inside string starting with line '
+                                           + row_num)
+                                    raise Exception(msg)
+                                raise
+                except StopIteration:
+                    lines.extend(new_rows)
+                    if len(lines) == 0:
+                        raise
+                self.pos += len(new_rows)
 
-        self.buf = []
+            self.buf = []
+        else:
+            lines = new_rows
 
         if self.skip_footer:
             lines = lines[:-self.skip_footer]


Test patch:
diff --git a/pandas/io/tests/test_parsers.py b/pandas/io/tests/test_parsers.py
--- a/pandas/io/tests/test_parsers.py
+++ b/pandas/io/tests/test_parsers.py
@@ -1037,6 +1037,24 @@ def test_iterator(self):
                                   iterator=True)
         self.assert_(isinstance(treader, TextFileReader))
 
+        # stopping iteration when on chunksize is specified, GH 3967
+        data = """A,B,C
+foo,1,2,3
+bar,4,5,6
+baz,7,8,9
+"""
+        reader = self.read_csv(StringIO(data), iterator=True)
+        result = list(reader)
+        expected = DataFrame(dict(A = [1,4,7], B = [2,5,8], C = [3,6,9]), index=['foo','bar','baz'])
+        tm.assert_frame_equal(result[0], expected)
+
+        # chunksize = 1
+        reader = self.read_csv(StringIO(data), chunksize=1)
+        result = list(reader)
+        expected = DataFrame(dict(A = [1,4,7], B = [2,5,8], C = [3,6,9]), index=['foo','bar','baz'])
+        self.assert_(len(result) == 3)
+        tm.assert_frame_equal(pd.concat(result), expected)
+
     def test_header_not_first_line(self):
         data = """got,to,ignore,this,line
 got,to,ignore,this,line