Repository: pandas-dev/pandas

Issue:
Pandas conversion of Timedelta is very slow
#### Code Sample, a copy-pastable example if possible

```python

a=pd.Series([pd.Timedelta(days=x) for x in np.random.randint(0, 10, 60000)])

# Let's convert Timedelta to days with Pandas
%time a.dt.days
# CPU times: user 457 ms, sys: 4.08 ms, total: 461 ms
# Wall time: 464 ms

# Let's convert Timedelta to days by division 
%time (a / np.timedelta64(1, 'D')).astype(np.int64)
# CPU times: user 3.19 ms, sys: 1.79 ms, total: 4.98 ms
# Wall time: 3.18 ms

# Make sure results are the same
((a / np.timedelta64(1, 'D')).astype(np.int64)==a.dt.days).value_counts()
# True    60000
# dtype: int64

```
#### Problem description

For large Series it takes very long time to do a simple conversion, this should be optimised.

#### Expected Output

.dt.days should be as quick as dividing by np.timedelta64(1, 'D')

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None

pandas: 0.20.3
pytest: None
pip: 9.0.1
setuptools: 36.4.0
Cython: 0.26
numpy: 1.13.1
scipy: 0.19.1
xarray: None
IPython: 6.1.0
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: None
tables: 3.4.2
numexpr: 2.6.4
feather: None
matplotlib: 2.0.2
openpyxl: None
xlrd: 1.1.0
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.9999999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None

</details>



Discussion:
so the issue is that https://github.com/pandas-dev/pandas/blob/master/pandas/core/indexes/timedeltas.py#L383 is not done in a vectorized way, IOW needs to simply construct the returned arrays and then use ``_maybe_mask_results``, rather than doing this in a list-comp with individual Timedelta construction.

you can't actually divide by ``np.timedelta64`` (well strictly speaking for days you can but not for any other unit), so rather fix the general problem.

want to have a go at a PR?
I took a look at this and re-factored the _get_field function to look as follows:

    def _get_field(self, m):
        def map_attr(x):
            nonlocal m
            return getattr(Timedelta(x), m)
        
        values = self.asi8
        hasnans = self.hasnans
        if hasnans:
            vfunc = np.vectorize(map_attr, otypes=[np.float])
        else:
            vfunc = np.vectorize(map_attr, otypes=[np.int])
        
        result = vfunc(values)
            
        return Index(result, name=self.name)

However, I didn't really see any tangible performance improvement. The [np.vectorize docs](https://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.vectorize.html) mention that the function is for convenience and not necessarily performance, as it's essentially a for loop. 

@jreback - is the code above that I provided in line with what you were expecting? If that's the case I'm not sure that is really the root cause from my initial tests
no this is not vectorization, see how other things are done in datetimes.py
Thanks Jeff. I refactored to the below and did see significant speed improvements:

    def _get_field(self, m):
        freqs = {
            'days' : 'D',
            'seconds' : 's',
            'microseconds' : 'us',
            'nanoseconds' : 'ns'
            }
        freq = freqs[m]
        
        values = self.asi8
        hasnans = self.hasnans
        result = libts.get_date_field(values, freq)
        if hasnans:
            result = self._maybe_mask_results(result, convert='float64')
        
        return Index(result, name=self.name)

Do you have a point of view as to where to the put the freqs dict I have above? I've included it in the method here for visibility, but I was thinking it could be better served as a classmethod to map the properties to their appropriate frequency codes
@Stanpol great!

you shouldn't need the freq dict, simply pass in the correct value in the accessor itself (which I think is what it was doing). and pls add some asv's for this.
Thanks for the tip on the accessors - that's easy enough. One issue I'm seeing now though is that I might need to be careful with handling dates vs time deltas. I noticed the test_fields method in test_timedelta.py is failing with the below:

    E       Index values are different (100.0 %)
    E       [left]:  Int64Index([2, 3, 4, 5, 6, 7, 8, 9, 10, 11], dtype='int64')
    E       [right]: Int64Index([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], dtype='int64')

The difference of one day I assume is due to me using the get_date_field method in libts while passing in Timedelta objects. Any tips on how to best handle that?

I'll take a look at adding some asv's as you suggest
what does this have to do with dates? 
you shouldn't be using this ``get_date_field``, you are *solely* calling routings on timedeltas.
My mistake. Looking further into this I noticed that all of the logic for converting between days, hours, minutes, seconds, etc... is contained within the `_ensure_components` method of the `_Timedelta` class in timedeltas.pyx, so it's difficult to avoid some kind of individual Timedelta construction as the code currently stands. 

I'll keep plugging at it but I haven't done much in C / Cython before so it may be slow going on my end to figure out how to make timedelta field access work similar to date objects. If anyone else out there has thoughts on how to tackle then by all means.

FWIW here's what I tried to implement in fields.pyx to mimic what exists for dates. I did this solely to check performance so there isn't any error handling. My very un-scientific tests weren't showing any improvement over existing code. 

    @cython.wraparound(False)
    @cython.boundscheck(False)
    def get_timedelta_field(ndarray[int64_t] tdindex, object field):
        cdef:
            Py_ssize_t i, count = 0
            ndarray[int32_t] out

        count = len(tdindex)
        out = np.empty(count, dtype='i4')

        for i in range(count):
            if tdindex[i] == NPY_NAT:
                out[i] = -1
                continue

            out[i] = getattr(_Timedelta(microseconds = tdindex[i] / 1000), field)
        
      return out
note that https://github.com/pandas-dev/pandas/pull/18161 was merged so the code you are working is slightly moved around but substantially the same

Resolving patch:
diff --git a/asv_bench/benchmarks/timedelta.py b/asv_bench/benchmarks/timedelta.py
--- a/asv_bench/benchmarks/timedelta.py
+++ b/asv_bench/benchmarks/timedelta.py
@@ -40,3 +40,46 @@ def setup(self):
 
     def test_add_td_ts(self):
         self.td + self.ts
+
+
+class TimedeltaProperties(object):
+    goal_time = 0.2
+
+    def setup(self):
+        self.td = Timedelta(days=365, minutes=35, seconds=25, milliseconds=35)
+
+    def time_timedelta_days(self):
+        self.td.days
+
+    def time_timedelta_seconds(self):
+        self.td.seconds
+
+    def time_timedelta_microseconds(self):
+        self.td.microseconds
+
+    def time_timedelta_nanoseconds(self):
+        self.td.nanoseconds
+
+
+class DatetimeAccessor(object):
+    goal_time = 0.2
+
+    def setup(self):
+        self.N = 100000
+        self.series = pd.Series(
+            pd.timedelta_range('1 days', periods=self.N, freq='h')
+        )
+    def time_dt_accessor(self):
+        self.series.dt
+
+    def time_timedelta_dt_accessor_days(self):
+        self.series.dt.days
+
+    def time_timedelta_dt_accessor_seconds(self):
+        self.series.dt.seconds
+
+    def time_timedelta_dt_accessor_microseconds(self):
+        self.series.dt.microseconds
+
+    def time_timedelta_dt_accessor_nanoseconds(self):
+        self.series.dt.nanoseconds
diff --git a/pandas/core/indexes/timedeltas.py b/pandas/core/indexes/timedeltas.py
--- a/pandas/core/indexes/timedeltas.py
+++ b/pandas/core/indexes/timedeltas.py
@@ -35,20 +35,15 @@
 from pandas._libs import (lib, index as libindex, tslib as libts,
                           join as libjoin, Timedelta, NaT, iNaT)
 from pandas._libs.tslibs.timedeltas import array_to_timedelta64
+from pandas._libs.tslibs.fields import get_timedelta_field
 
 
 def _field_accessor(name, alias, docstring=None):
     def f(self):
+        values = self.asi8
+        result = get_timedelta_field(values, alias)
         if self.hasnans:
-            result = np.empty(len(self), dtype='float64')
-            mask = self._isnan
-            imask = ~mask
-            result.flat[imask] = np.array([getattr(Timedelta(val), alias)
-                                           for val in self.asi8[imask]])
-            result[mask] = np.nan
-        else:
-            result = np.array([getattr(Timedelta(val), alias)
-                               for val in self.asi8], dtype='int64')
+            result = self._maybe_mask_results(result, convert='float64')
 
         return Index(result, name=self.name)
 


Test patch: