Repository: pandas-dev/pandas

Issue:
PERF: perf regressions vs 0.14.0
We have many + perf fixes in 0.14.1 yeh!

but need to look at these (resamples could be spurious)

```
dataframe_resample_mean_string               |   4.0483 |   2.4170 |   1.6749 |
dataframe_resample_max_numpy                 |   3.3394 |   1.7230 |   1.9381 |
dataframe_resample_max_string                |   3.3154 |   1.7047 |   1.9448 |
dataframe_resample_min_numpy                 |   3.3020 |   1.6843 |   1.9604 |
dataframe_resample_min_string                |   3.2830 |   1.6587 |   1.9793 |
timeseries_timestamp_downsample_mean         |  11.4803 |   4.0847 |   2.8106 |
timeseries_period_downsample_mean            |  29.7631 |  10.3510 |   2.8754 |
datetimeindex_normalize                      |  77.5570 |   3.0633 |  25.3183 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [272dae5] : BUG: doc example in groupby.rst (GH7559 / GH7628)
Base   [da0f7ae] : RLS: 0.14.0 final
```



Discussion:
cc @sinhrks

There are some pretty hefty hits for `DatetimeIndex`-ed repring, so much so that I have to `KeyboardInterrupt` the repring. I'll see if I can't track this down. It looks like there's some unnecessary vectorization of timezone manipulation.

can you post your example?

```
In [1]: period = 1.0 / 2048 * 1e9

In [2]: freq = pd.datetools.Nano(period)

In [3]: df = DataFrame({'a': np.random.randn(6e6)}, index=pd.date_range('now', periods=6e6, freq=freq, tz='EST'))

In [4]: df.a  # takes a very long time
```

hm maybe repr-ing is not the issue `repr(df.a)` finishes in a timely manner.

what does ipython actually do after executing, maybe it caches it or something?

the answer is in the  format, while I ctrl-c....its in `trim_zeros` (its prob trimming on the entire frame and not on the displayed portion)

hm when i control-c it's in a `np.vectorize`-d function. vectorize is not really that much faster than just map + lambda which i think is the issue

okay maybe not ... just cython'd that and it didn't improve the situation

that _is_ where the bottleneck is though

```
In [4]: df.a
> /home/phillip/Documents/code/py/pandas/pandas/tseries/frequencies.py(674)__init__()
    673             import ipdb; ipdb.set_trace()
--> 674             self.values = tslib.tz_convert_safe_dst(self.values, 'UTC',
    675                                                     index.tz)

ipdb> n
> /home/phillip/Documents/code/py/pandas/pandas/tseries/frequencies.py(675)__init__()
    674             self.values = tslib.tz_convert_safe_dst(self.values, 'UTC',
--> 675                                                     index.tz)
    676             #self.values = np.vectorize(f)(self.values)

ipdb> n  # bueller ..... bueller ..... anybody
```

yes...that is the same problem as in the vbench above

ok

what does bcdc6d98 actually fix? i don't see any docs on it

If you put the commented code back this fixes the entire problem: https://github.com/pydata/pandas/blob/master/pandas/tseries/frequencies.py#L675

yep

can you figure out where that was changed?

the commit i just put up is where it was changed

``` sh
git blame -L 671,677 pandas/tseries/frequencies.py
```

cc @sinhrks https://github.com/pydata/pandas/commit/bcdc6d98d0757f5ec6557ae6565cc64e309b0cc6

partially closd by #7652 

@cpcloud this also fixes the display issue from above (though no idea how to actully test it)

cc @sinhrks. Seems `min/max` are doing a vair amount of computation on an Index w/o missing values. This is why the resamples are now slower. Can you take a look?

Yes, will check.

It looks offset ops get also little slower. I'll take a look.

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
frame_ctor_dtindex_Millix2                   |   2.3377 |   1.8246 |   1.2812 |
frame_ctor_dtindex_YearEndx2                 |   2.7140 |   2.1173 |   1.2818 |
frame_ctor_dtindex_Microx2                   |   2.3347 |   1.8183 |   1.2840 |
frame_ctor_dtindex_Hourx2                    |   2.3340 |   1.8170 |   1.2846 |
frame_ctor_dtindex_Microx1                   |   2.3340 |   1.8164 |   1.2850 |
frame_ctor_dtindex_Dayx1                     |   2.3523 |   1.8146 |   1.2963 |
...
timeseries_day_incr                          |   0.0190 |   0.0054 |   3.5147 |
timeseries_day_apply                         |   0.0173 |   0.0033 |   5.1905 |
```


Resolving patch:
diff --git a/pandas/core/base.py b/pandas/core/base.py
--- a/pandas/core/base.py
+++ b/pandas/core/base.py
@@ -6,7 +6,7 @@
 from pandas.core import common as com
 import pandas.core.nanops as nanops
 import pandas.tslib as tslib
-
+from pandas.util.decorators import cache_readonly
 
 class StringMixin(object):
 
@@ -392,6 +392,11 @@ def _box_values(self, values):
         import pandas.lib as lib
         return lib.map_infer(values, self._box_func)
 
+    @cache_readonly
+    def hasnans(self):
+        """ return if I have any nans; enables various perf speedups """
+        return (self.asi8 == tslib.iNaT).any()
+
     @property
     def asobject(self):
         from pandas.core.index import Index
@@ -408,11 +413,18 @@ def min(self, axis=None):
         Overridden ndarray.min to return an object
         """
         try:
-            mask = self.asi8 == tslib.iNaT
-            if mask.any():
+            i8 = self.asi8
+
+            # quick check
+            if len(i8) and self.is_monotonic:
+                if i8[0] != tslib.iNaT:
+                    return self._box_func(i8[0])
+
+            if self.hasnans:
+                mask = i8 == tslib.iNaT
                 min_stamp = self[~mask].asi8.min()
             else:
-                min_stamp = self.asi8.min()
+                min_stamp = i8.min()
             return self._box_func(min_stamp)
         except ValueError:
             return self._na_value
@@ -422,11 +434,18 @@ def max(self, axis=None):
         Overridden ndarray.max to return an object
         """
         try:
-            mask = self.asi8 == tslib.iNaT
-            if mask.any():
+            i8 = self.asi8
+
+            # quick check
+            if len(i8) and self.is_monotonic:
+                if i8[-1] != tslib.iNaT:
+                    return self._box_func(i8[-1])
+
+            if self.hasnans:
+                mask = i8 == tslib.iNaT
                 max_stamp = self[~mask].asi8.max()
             else:
-                max_stamp = self.asi8.max()
+                max_stamp = i8.max()
             return self._box_func(max_stamp)
         except ValueError:
             return self._na_value
diff --git a/pandas/core/index.py b/pandas/core/index.py
--- a/pandas/core/index.py
+++ b/pandas/core/index.py
@@ -2072,7 +2072,7 @@ def __contains__(self, other):
 
         try:
             # if other is a sequence this throws a ValueError
-            return np.isnan(other) and self._hasnans
+            return np.isnan(other) and self.hasnans
         except ValueError:
             try:
                 return len(other) <= 1 and _try_get_item(other) in self
@@ -2109,7 +2109,7 @@ def _isnan(self):
         return np.isnan(self.values)
 
     @cache_readonly
-    def _hasnans(self):
+    def hasnans(self):
         return self._isnan.any()
 
     @cache_readonly
diff --git a/pandas/tseries/resample.py b/pandas/tseries/resample.py
--- a/pandas/tseries/resample.py
+++ b/pandas/tseries/resample.py
@@ -174,7 +174,7 @@ def _get_time_bins(self, ax):
         binner, bin_edges = self._adjust_bin_edges(binner, ax_values)
 
         # general version, knowing nothing about relative frequencies
-        bins = lib.generate_bins_dt64(ax_values, bin_edges, self.closed)
+        bins = lib.generate_bins_dt64(ax_values, bin_edges, self.closed, hasnans=ax.hasnans)
 
         if self.closed == 'right':
             labels = binner
@@ -188,7 +188,7 @@ def _get_time_bins(self, ax):
             elif not trimmed:
                 labels = labels[:-1]
 
-        if (ax_values == tslib.iNaT).any():
+        if ax.hasnans:
             binner = binner.insert(0, tslib.NaT)
             labels = labels.insert(0, tslib.NaT)
 


Test patch: