Repository: pandas-dev/pandas

Issue:
df.dtypes.values is not O(1) and repr(df) is therefore slow for large frames
For the FEC dataset, it takes about 1.5 sec to get a repr. and prun
puts it all in `infer_dtype`.



Discussion:
do you have a link to the dataset...can't seem to find mine

ftp://ftp.fec.gov/FEC/Presidential_Map/2012/P00000001/P00000001-ALL.zip

I believe that this is the problem.

It is trying to see if their are floats in an object array. I would simply not do this at all
or short-circuit it.

```
Breakpoint 2 at /mnt/home/jreback/pandas/pandas/core/format.py:1663
(Pdb) c
> /mnt/home/jreback/pandas/pandas/core/format.py(1663)_format_strings()
-> is_float = lib.map_infer(vals, com.is_float) & notnull(vals)
(Pdb) l
1658                    # object dtype
1659                    return '%s' % formatter(x)
1660 
1661            vals = self.values
1662 
1663B->         is_float = lib.map_infer(vals, com.is_float) & notnull(vals)
1664            leading_space = is_float.any()
1665 
1666            fmt_values = []
1667            for i, v in enumerate(vals):
1668                if not is_float[i] and leading_space:
```

It's probably there to support the `float_format` arg of `to_string`. I'll have to think about it.

for an object dtype you could warn if it 'looks' like float, but otherwise skip it
its checking strings that aren't numbers at all is the problem

isn't `looks_like_float()` exactly what `map_infer` does? how could it be faster if
I have to check each value for "appearence"?

It doesn't need to do this for values not displayed in the output. that's it.

right!

That's not where the bottleneck is.
What's this?

```
In [10]: %timeit df.dtypes.values
1 loops, best of 3: 178 ms per loop
```

aren't dtypes just a lookup?

this issue address this, but it needs reworking to make it more internal as I havev indicated: https://github.com/pydata/pandas/pull/5740

Related (djeavu): 3cb6961, https://github.com/pydata/pandas/issues/2807#issuecomment-13931051

I have got a PR...give me a few

There's an off chance this might be the cause of a lot of the slowdowns we saw in 0.13
after the NDFrame refactor. is the change in behaviour related? if yes, hurrah.

anything with `df.apply` in it is generally bad (as dsm fixed for str.extract), when needed internally

https://github.com/pydata/pandas/issues/5660

frame_get_dtype_counts                       |   0.1843 |   0.1113 |   1.6552 |

<del>less then what I expected, but should have sent bells ringing.</del>

Unrelated in fact.

@jreback: dsm->unutbu.   Can't take credit for that one. :^)

@dsm054 sorry....you are right!! morning confusion

That only cuts it in half. Is this expected?

```
df=pd.read_csv('P00000001-ALL.csv',low_memory=False)
%timeit df.iloc[:100, 4]
10 loops, best of 3: 88.1 ms per loop
```

Isn't slicing supposed to be cheap?

let me look


Resolving patch:
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -1889,7 +1889,10 @@ def make_block(values, items, ref_items, klass=None, ndim=None, dtype=None,
 
             if np.prod(values.shape):
                 flat = values.ravel()
-                inferred_type = lib.infer_dtype(flat)
+
+                # try with just the first element; we just need to see if
+                # this is a datetime or not
+                inferred_type = lib.infer_dtype(flat[0:1])
                 if inferred_type in ['datetime', 'datetime64']:
 
                     # we have an object array that has been inferred as
diff --git a/vb_suite/indexing.py b/vb_suite/indexing.py
--- a/vb_suite/indexing.py
+++ b/vb_suite/indexing.py
@@ -167,3 +167,10 @@
 
 frame_loc_dups = Benchmark('df2.loc[idx]', setup,
                             start_date=datetime(2013, 1, 1))
+
+setup = common_setup + """
+df = DataFrame(dict( A = [ 'foo'] * 1000000))
+"""
+
+frame_iloc_big = Benchmark('df.iloc[:100,0]', setup,
+                            start_date=datetime(2013, 1, 1))


Test patch:
diff --git a/pandas/io/tests/test_stata.py b/pandas/io/tests/test_stata.py
--- a/pandas/io/tests/test_stata.py
+++ b/pandas/io/tests/test_stata.py
@@ -4,9 +4,12 @@
 import os
 import warnings
 import nose
+import sys
+from distutils.version import LooseVersion
 
 import numpy as np
 
+import pandas as pd
 from pandas.core.frame import DataFrame, Series
 from pandas.io.parsers import read_csv
 from pandas.io.stata import read_stata, StataReader
@@ -66,6 +69,9 @@ def test_read_dta1(self):
         tm.assert_frame_equal(parsed_13, expected)
 
     def test_read_dta2(self):
+        if LooseVersion(sys.version) < '2.7':
+            raise nose.SkipTest('datetime interp under 2.6 is faulty')
+
         expected = DataFrame.from_records(
             [
                 (
@@ -89,14 +95,14 @@ def test_read_dta2(self):
                     datetime(2, 1, 1)
                 ),
                 (
-                    np.datetime64('NaT'),
-                    np.datetime64('NaT'),
-                    np.datetime64('NaT'),
-                    np.datetime64('NaT'),
-                    np.datetime64('NaT'),
-                    np.datetime64('NaT'),
-                    np.datetime64('NaT'),
-                    np.datetime64('NaT')
+                    pd.NaT,
+                    pd.NaT,
+                    pd.NaT,
+                    pd.NaT,
+                    pd.NaT,
+                    pd.NaT,
+                    pd.NaT,
+                    pd.NaT,
                 )
             ],
             columns=['datetime_c', 'datetime_big_c', 'date', 'weekly_date',
diff --git a/pandas/tseries/tests/test_timeseries.py b/pandas/tseries/tests/test_timeseries.py
--- a/pandas/tseries/tests/test_timeseries.py
+++ b/pandas/tseries/tests/test_timeseries.py
@@ -829,6 +829,11 @@ def test_to_datetime_mixed(self):
         expected = Series([NaT,Timestamp('20130408'),Timestamp('20130409')])
         assert_series_equal(result,expected)
 
+        # mixed datetime/np.datetime64('NaT')
+        result = Series(to_datetime([dt.datetime(2000,1,1),np.datetime64('NaT')]))
+        expected = Series([dt.datetime(2000,1,1),NaT])
+        assert_series_equal(result, expected)
+
     def test_dayfirst(self):
 
         # GH 3341