Repository: pandas-dev/pandas

Issue:
pd.Series.asof performance regression: 14x slower
#### pd.Series.asof takes 14x longer in 0.19.0 than 0.18.1

``` python
In[14]: pd.__version__
Out[14]: '0.18.1'
In[15]: s = pd.Series([1, 1, 1, np.nan, np.nan])
In[16]: %timeit s.asof(4)
The slowest run took 8.25 times longer than the fastest. This could mean that an intermediate result is being cached.
100000 loops, best of 3: 8.96 µs per loop

In[8]: pd.__version__
Out[8]: '0.19.0'
In[9]: s = pd.Series([1, 1, 1, np.nan, np.nan])
In[10]: %timeit s.asof(4)
10000 loops, best of 3: 125 µs per loop
```



Discussion:
Yes, I have noticed that in our benchmarks as well, but forgot to open an issue. Thanks for reporting! 
If you want to have a look what could be the cause, very welcome!

well < 0.19.0 was pretty much broken for non-monotonic indexes and when nulls existed. so you get correct or you can have a tiny differential in perf.

Not sure how this actually matters anyhow except if you are calling this in a loop, which is completely non-idiomatic.

> Not sure how this actually matters anyhow except if you are calling this in a loop, which is completely non-idiomatic.

pd.Series.asof is extremely flexible. If I pass in a value before the start of the index, it returns NA. If I pass in a value after the end of the index, it returns the very last non-NA value.

Now in one of my cases I have several thousand time series and for a given date, I want to get the asof value for each one and I currently do this in a loop. The date can be anything like a weekend or a holiday.

I've thought about creating a giant DataFrame, reindex it to include every single calendar date, then fillna(method='ffill) but I still need to deal with the dates before the start or after the end of the index with if statements.

And if those time series cover different date/time ranges, the resulting DataFrame is extremely sparse.

Maybe there is a better and more idiomatic way for this particular case?

In other cases, asof has to be called on individual Series without the possibility of concatenating them into a giant DataFrame. The speed of the call is therefore very imporant.

So this is a general method which can handle both Series and DataFrames. an implementation that handles both directly in cython could certainly be done. I would encourage a pull-request to do this.

For example the nulls are pre-computed here, which makes the code much simpler, but is not necessary when iterating (which you can only do in a performant way in cython, comparing nulls as you go).

> well < 0.19.0 was pretty much broken for non-monotonic indexes and when nulls existed. so you get correct or you can have a tiny differential in perf.

Non-monotonic indices are not involved I think, as it just raises for that (the error raise is added in 0.19.0, but that is of course not the perf issue). I rather think this is a case where the generalization of the method for both Series and DataFrame has impacted the performance for (certain) Series only cases. 

With a few small adaptions in the current code I can bring it back to 0.18 performance for series (not calculating all nulls in advance + indexing the values instead of Series (in case of Series you return a scalar, so don't need to index the object itself with `iloc`))

@jorisvandenbossche the non-monotonic checks actually are important, they do take _some_ small amount of time. Again when iterating you can do these in-line.

@laudney would welcome a pull-request.

@laudney I think it would be rather simple to fix the performance issue. You have to compare the implemention of Series.asof of 0.18 with the current implementation (the differences are the pre-computations of the nulls and no longer working with the underlying array but with the series itself. Both give a performance degradation). As @jreback said, pull request is very welcome!

@jreback @jorisvandenbossche Let me take a look but no promise though!


Resolving patch:
diff --git a/asv_bench/benchmarks/timeseries.py b/asv_bench/benchmarks/timeseries.py
--- a/asv_bench/benchmarks/timeseries.py
+++ b/asv_bench/benchmarks/timeseries.py
@@ -284,56 +284,77 @@ class timeseries_asof(object):
     goal_time = 0.2
 
     def setup(self):
-        self.N = 100000
-        self.rng = date_range(start='1/1/2000', periods=self.N, freq='T')
-        if hasattr(Series, 'convert'):
-            Series.resample = Series.convert
-        self.ts = Series(np.random.randn(self.N), index=self.rng)
         self.N = 10000
         self.rng = date_range(start='1/1/1990', periods=self.N, freq='53s')
-        self.ts = Series(np.random.randn(self.N), index=self.rng)
         self.dates = date_range(start='1/1/1990', periods=(self.N * 10), freq='5s')
+        self.ts = Series(np.random.randn(self.N), index=self.rng)
+        self.ts2 = self.ts.copy()
+        self.ts2[250:5000] = np.nan
+        self.ts3 = self.ts.copy()
+        self.ts3[-5000:] = np.nan
 
-    def time_timeseries_asof(self):
+    # test speed of pre-computing NAs.
+    def time_asof_list(self):
         self.ts.asof(self.dates)
 
+    # should be roughly the same as above.
+    def time_asof_nan_list(self):
+        self.ts2.asof(self.dates)
 
-class timeseries_asof_nan(object):
-    goal_time = 0.2
+    # test speed of the code path for a scalar index
+    # without *while* loop
+    def time_asof_single(self):
+        self.ts.asof(self.dates[0])
 
-    def setup(self):
-        self.N = 100000
-        self.rng = date_range(start='1/1/2000', periods=self.N, freq='T')
-        if hasattr(Series, 'convert'):
-            Series.resample = Series.convert
-        self.ts = Series(np.random.randn(self.N), index=self.rng)
-        self.N = 10000
-        self.rng = date_range(start='1/1/1990', periods=self.N, freq='53s')
-        self.ts = Series(np.random.randn(self.N), index=self.rng)
-        self.dates = date_range(start='1/1/1990', periods=(self.N * 10), freq='5s')
-        self.ts[250:5000] = np.nan
+    # test speed of the code path for a scalar index
+    # before the start. should be the same as above.
+    def time_asof_single_early(self):
+        self.ts.asof(self.dates[0] - dt.timedelta(10))
 
-    def time_timeseries_asof_nan(self):
-        self.ts.asof(self.dates)
+    # test the speed of the code path for a scalar index
+    # with a long *while* loop. should still be much
+    # faster than pre-computing all the NAs.
+    def time_asof_nan_single(self):
+        self.ts3.asof(self.dates[-1])
 
 
-class timeseries_asof_single(object):
+class timeseries_dataframe_asof(object):
     goal_time = 0.2
 
     def setup(self):
-        self.N = 100000
-        self.rng = date_range(start='1/1/2000', periods=self.N, freq='T')
-        if hasattr(Series, 'convert'):
-            Series.resample = Series.convert
-        self.ts = Series(np.random.randn(self.N), index=self.rng)
         self.N = 10000
+        self.M = 100
         self.rng = date_range(start='1/1/1990', periods=self.N, freq='53s')
-        self.ts = Series(np.random.randn(self.N), index=self.rng)
         self.dates = date_range(start='1/1/1990', periods=(self.N * 10), freq='5s')
+        self.ts = DataFrame(np.random.randn(self.N, self.M), index=self.rng)
+        self.ts2 = self.ts.copy()
+        self.ts2.iloc[250:5000] = np.nan
+        self.ts3 = self.ts.copy()
+        self.ts3.iloc[-5000:] = np.nan
+
+    # test speed of pre-computing NAs.
+    def time_asof_list(self):
+        self.ts.asof(self.dates)
 
-    def time_timeseries_asof_single(self):
+    # should be roughly the same as above.
+    def time_asof_nan_list(self):
+        self.ts2.asof(self.dates)
+
+    # test speed of the code path for a scalar index
+    # with pre-computing all NAs.
+    def time_asof_single(self):
         self.ts.asof(self.dates[0])
 
+    # should be roughly the same as above.
+    def time_asof_nan_single(self):
+        self.ts3.asof(self.dates[-1])
+
+    # test speed of the code path for a scalar index
+    # before the start. should be without the cost of
+    # pre-computing all the NAs.
+    def time_asof_single_early(self):
+        self.ts.asof(self.dates[0] - dt.timedelta(10))
+
 
 class timeseries_custom_bday_apply(object):
     goal_time = 0.2
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -3735,10 +3735,10 @@ def asof(self, where, subset=None):
         if not self.index.is_monotonic:
             raise ValueError("asof requires a sorted index")
 
-        if isinstance(self, ABCSeries):
+        is_series = isinstance(self, ABCSeries)
+        if is_series:
             if subset is not None:
                 raise ValueError("subset is not valid for Series")
-            nulls = self.isnull()
         elif self.ndim > 2:
             raise NotImplementedError("asof is not implemented "
                                       "for {type}".format(type(self)))
@@ -3747,9 +3747,9 @@ def asof(self, where, subset=None):
                 subset = self.columns
             if not is_list_like(subset):
                 subset = [subset]
-            nulls = self[subset].isnull().any(1)
 
-        if not is_list_like(where):
+        is_list = is_list_like(where)
+        if not is_list:
             start = self.index[0]
             if isinstance(self.index, PeriodIndex):
                 where = Period(where, freq=self.index.freq).ordinal
@@ -3758,16 +3758,26 @@ def asof(self, where, subset=None):
             if where < start:
                 return np.nan
 
-            loc = self.index.searchsorted(where, side='right')
-            if loc > 0:
-                loc -= 1
-            while nulls[loc] and loc > 0:
-                loc -= 1
-            return self.iloc[loc]
+            # It's always much faster to use a *while* loop here for
+            # Series than pre-computing all the NAs. However a
+            # *while* loop is extremely expensive for DataFrame
+            # so we later pre-compute all the NAs and use the same
+            # code path whether *where* is a scalar or list.
+            # See PR: https://github.com/pandas-dev/pandas/pull/14476
+            if is_series:
+                loc = self.index.searchsorted(where, side='right')
+                if loc > 0:
+                    loc -= 1
+
+                values = self._values
+                while loc > 0 and isnull(values[loc]):
+                    loc -= 1
+                return values[loc]
 
         if not isinstance(where, Index):
-            where = Index(where)
+            where = Index(where) if is_list else Index([where])
 
+        nulls = self.isnull() if is_series else self[subset].isnull().any(1)
         locs = self.index.asof_locs(where, ~(nulls.values))
 
         # mask the missing
@@ -3775,7 +3785,7 @@ def asof(self, where, subset=None):
         data = self.take(locs, is_copy=False)
         data.index = where
         data.loc[missing] = np.nan
-        return data
+        return data if is_list else data.iloc[-1]
 
     # ----------------------------------------------------------------------
     # Action Methods


Test patch: