Repository: pandas-dev/pandas

Issue:
Make itertuples really an iterator/generator in implementation, not just return type
`itertuples` is not really an iterator/generator and constructs a copy of whole DataFrame in memory. Ideally it would return just an iterator and construct row by row as it is being iterated over.


Discussion:
looks like a generator to me

```
In [1]: df = pd.DataFrame({'A': range(3), 'B': list('ABC')})

In [2]: df.itertuples()
Out[2]: <map at 0x10922c080>

In [3]: list(df.itertuples())
Out[3]: 
[Pandas(Index=0, A=0, B='A'),
 Pandas(Index=1, A=1, B='B'),
 Pandas(Index=2, A=2, B='C')]

In [5]: i = df.itertuples()

In [6]: next(i)
Out[6]: Pandas(Index=0, A=0, B='A')

In [7]: next(i)
Out[7]: Pandas(Index=1, A=1, B='B')

In [8]: next(i)
Out[8]: Pandas(Index=2, A=2, B='C')

In [9]: next(i)
---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
<ipython-input-9-a883b34d6d8a> in <module>()
----> 1 next(i)

StopIteration: 
```
That's just because you return a `zip`. But inside the function, you create whole list of values first. See [here](https://github.com/pandas-dev/pandas/blob/3e691a4cba566472bef03ef9bbaec701498670e1/pandas/core/frame.py#L826):

```
arrays.extend(self.iloc[:, k] for k in range(len(self.columns)))
```

It looks like iterator because of `zip(*arrays)`, but `arrays` is not an iterator. This is a problem.
You can test the issue here by doing:

```python
d = pandas.DataFrame({'a': range(100000000)})
for a in d.itertuples(index=False, name=None):
    print(a)
```

Do this in Python interpreter. Note the time it takes to create `d`. Now, when you press enter after `print` line, twice, note the time it takes to start producing anything. It takes this time because it is first creating all rows in memory, before iterating over them.
and if u want to fix it pls submit a PR
this is truly an iterator 

so what u describe is an implementation detail 
Implementation detail which blows up memory and performance?

Anything can be made look like iterator. But if does not really behave like iterator, it is not an iterator.

I think this is a bug. Please reopen this. And then me or somebody else can make a pull request.
This goes even deeper. Also iterating over a series constructs a list internally:

```python
d = pandas.DataFrame({'a': range(100000000)})
for a in d['a']:
    print(a)
```

This will also have a large delay before starting sending results back.
I agree that ideally it would be more lazy the iteration (although not really a priority issue for me), and since we would accept a PR to fix, let's keep the issue open.
@mitar you are welcome to submit a PR, however, this method follows exactly the pandas paradigm. We create a new copy of things then hand it back to you, here the handing back is an iterator. If you can optimize this, great, but you are fighting standard practice. Further you may slow things down by doing this in the common case.
This would be a welcome fix if possible.

@mitar a couple things to watch out for, which we just hit with `Categorical.__iter__`:

- Scalars have to be converted from NumPy scalars to Python scalars
- We need to avoid tons of calls to `Series/Frame.__getitem__`, as this is relatively slow
I made: #20796
> We need to avoid tons of calls to `Series/Frame.__getitem__`, as this is relatively slow

Calling `iloc` is OK? Or is this the same slow as `__getitem__`?
I think that the PR #20796 is ready to be reviewed.

Resolving patch:
diff --git a/asv_bench/benchmarks/frame_methods.py b/asv_bench/benchmarks/frame_methods.py
--- a/asv_bench/benchmarks/frame_methods.py
+++ b/asv_bench/benchmarks/frame_methods.py
@@ -103,6 +103,7 @@ def setup(self):
         self.df2 = DataFrame(np.random.randn(N * 50, 10))
         self.df3 = DataFrame(np.random.randn(N, 5 * N),
                              columns=['C' + str(c) for c in range(N * 5)])
+        self.df4 = DataFrame(np.random.randn(N * 1000, 10))
 
     def time_iteritems(self):
         # (monitor no-copying behaviour)
@@ -119,10 +120,70 @@ def time_iteritems_indexing(self):
         for col in self.df3:
             self.df3[col]
 
+    def time_itertuples_start(self):
+        self.df4.itertuples()
+
+    def time_itertuples_read_first(self):
+        next(self.df4.itertuples())
+
     def time_itertuples(self):
-        for row in self.df2.itertuples():
+        for row in self.df4.itertuples():
+            pass
+
+    def time_itertuples_to_list(self):
+        list(self.df4.itertuples())
+
+    def mem_itertuples_start(self):
+        return self.df4.itertuples()
+
+    def peakmem_itertuples_start(self):
+        self.df4.itertuples()
+
+    def mem_itertuples_read_first(self):
+        return next(self.df4.itertuples())
+
+    def peakmem_itertuples(self):
+        for row in self.df4.itertuples():
+            pass
+
+    def mem_itertuples_to_list(self):
+        return list(self.df4.itertuples())
+
+    def peakmem_itertuples_to_list(self):
+        list(self.df4.itertuples())
+
+    def time_itertuples_raw_start(self):
+        self.df4.itertuples(index=False, name=None)
+
+    def time_itertuples_raw_read_first(self):
+        next(self.df4.itertuples(index=False, name=None))
+
+    def time_itertuples_raw_tuples(self):
+        for row in self.df4.itertuples(index=False, name=None):
             pass
 
+    def time_itertuples_raw_tuples_to_list(self):
+        list(self.df4.itertuples(index=False, name=None))
+
+    def mem_itertuples_raw_start(self):
+        return self.df4.itertuples(index=False, name=None)
+
+    def peakmem_itertuples_raw_start(self):
+        self.df4.itertuples(index=False, name=None)
+
+    def peakmem_itertuples_raw_read_first(self):
+        next(self.df4.itertuples(index=False, name=None))
+
+    def peakmem_itertuples_raw(self):
+        for row in self.df4.itertuples(index=False, name=None):
+            pass
+
+    def mem_itertuples_raw_to_list(self):
+        return list(self.df4.itertuples(index=False, name=None))
+
+    def peakmem_itertuples_raw_to_list(self):
+        list(self.df4.itertuples(index=False, name=None))
+
     def time_iterrows(self):
         for row in self.df.iterrows():
             pass
diff --git a/pandas/core/base.py b/pandas/core/base.py
--- a/pandas/core/base.py
+++ b/pandas/core/base.py
@@ -8,7 +8,7 @@
 
 import pandas._libs.lib as lib
 import pandas.compat as compat
-from pandas.compat import PYPY, OrderedDict, builtins
+from pandas.compat import PYPY, OrderedDict, builtins, map, range
 from pandas.compat.numpy import function as nv
 from pandas.errors import AbstractMethodError
 from pandas.util._decorators import Appender, Substitution, cache_readonly
@@ -1072,7 +1072,13 @@ def __iter__(self):
         (for str, int, float) or a pandas scalar
         (for Timestamp/Timedelta/Interval/Period)
         """
-        return iter(self.tolist())
+        # We are explicity making element iterators.
+        if is_datetimelike(self._values):
+            return map(com.maybe_box_datetimelike, self._values)
+        elif is_extension_array_dtype(self._values):
+            return iter(self._values)
+        else:
+            return map(self._values.item, range(self._values.size))
 
     @cache_readonly
     def hasnans(self):
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -898,10 +898,10 @@ def itertuples(self, index=True, name="Pandas"):
         Animal(Index='hawk', num_legs=2, num_wings=2)
         """
         arrays = []
-        fields = []
+        fields = list(self.columns)
         if index:
             arrays.append(self.index)
-            fields.append("Index")
+            fields.insert(0, "Index")
 
         # use integer indexing because of possible duplicate column names
         arrays.extend(self.iloc[:, k] for k in range(len(self.columns)))
@@ -911,10 +911,9 @@ def itertuples(self, index=True, name="Pandas"):
         if name is not None and len(self.columns) + index < 256:
             # `rename` is unsupported in Python 2.6
             try:
-                itertuple = collections.namedtuple(name,
-                                                   fields + list(self.columns),
-                                                   rename=True)
+                itertuple = collections.namedtuple(name, fields, rename=True)
                 return map(itertuple._make, zip(*arrays))
+
             except Exception:
                 pass
 


Test patch: