Repository: pandas-dev/pandas

Issue:
PERF: improve to_timedelta perf for string-like
see #6754 
related #3009, #8190 
from [SO](http://stackoverflow.com/questions/30894267/defining-a-function-pandas)

convering string like timedeltas, e.g.

'1 days', '3 days, 00:00:03', '3 days 2 hours'

is slow because it requires a regex match for each element. Can put cythonized (like format parsing for datetimes), or since only very limited formats supported can simply test them (in cython) and convert.

BUG: indexing error when setting item in empty Series which has a frequency
Another somewhat degenerate case that works in 0.15.2, fails in 0.16.1

``` python
import pandas as pd

pd.show_versions()


ts = pd.TimeSeries()

ts[pd.datetime(2012, 1, 1)] = 47


ts2 = pd.TimeSeries(0, pd.date_range('2011-01-01', '2011-01-01'))[:0]

ts2[pd.datetime(2012, 1, 1)] = 47
```

0.15.2 output:

``` python
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.9.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 26 Stepping 5, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.15.2
nose: 1.3.4
Cython: 0.21
numpy: 1.9.2
scipy: 0.14.0
statsmodels: 0.5.0
IPython: 3.0.0-dev
sphinx: 1.2.3
patsy: 0.3.0
dateutil: 2.4.2
pytz: 2015.4
bottleneck: 0.8.0
tables: 3.1.1
numexpr: 2.3.1
matplotlib: 1.4.2
openpyxl: None
xlrd: 0.9.3
xlwt: None
xlsxwriter: 0.5.7
lxml: 3.4.0
bs4: 4.3.2
html5lib: 0.999
httplib2: None
apiclient: None
rpy2: None
sqlalchemy: 0.9.7
pymysql: None
psycopg2: None
```

0.16.1 output:

``` python
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.9.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 26 Stepping 5, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.16.1-46-g0aceb38
nose: 1.3.6
Cython: 0.22
numpy: 1.9.2
scipy: 0.14.0
statsmodels: 0.6.1
IPython: 3.1.0
sphinx: 1.3.1
patsy: 0.3.0
dateutil: 2.4.2
pytz: 2015.4
bottleneck: 0.8.0
tables: 3.1.1
numexpr: 2.3.1
matplotlib: 1.4.3
openpyxl: None
xlrd: 0.9.3
xlwt: None
xlsxwriter: 0.7.2
lxml: None
bs4: 4.3.2
html5lib: 0.999
httplib2: None
apiclient: None
sqlalchemy: 1.0.4
pymysql: None
psycopg2: None

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
c:\test_set_empty_series_with_freq.py in <module>()
     11 ts2 = pd.TimeSeries(0, pd.date_range('2011-01-01', '2011-01-01'))[:0]
     12
---> 13 ts2[pd.datetime(2012, 1, 1)] = 47

c:\python\envs\pandas-0.16.1\lib\site-packages\pandas\core\series.pyc in __setitem__(self, key, value)
    687         # do the setitem
    688         cacher_needs_updating = self._check_is_chained_assignment_possible()
--> 689         setitem(key, value)
    690         if cacher_needs_updating:
    691             self._maybe_update_cacher()

c:\python\envs\pandas-0.16.1\lib\site-packages\pandas\core\series.pyc in setitem(key, value)
    660                             pass
    661                 try:
--> 662                     self.loc[key] = value
    663                 except:
    664                     print ""

c:\python\envs\pandas-0.16.1\lib\site-packages\pandas\core\indexing.pyc in __setitem__(self, key, value)
    113     def __setitem__(self, key, value):
    114         indexer = self._get_setitem_indexer(key)
--> 115         self._setitem_with_indexer(indexer, value)
    116
    117     def _has_valid_type(self, k, axis):

c:\python\envs\pandas-0.16.1\lib\site-packages\pandas\core\indexing.pyc in _setitem_with_indexer(self, indexer, value)
    272                 if self.ndim == 1:
    273                     index = self.obj.index
--> 274                     new_index = index.insert(len(index),indexer)
    275
    276                     # this preserves dtype of the value

c:\python\envs\pandas-0.16.1\lib\site-packages\pandas\tseries\index.pyc in insert(self, loc, item)
   1523             # check freq can be preserved on edge cases
   1524             if self.freq is not None:
-> 1525                 if (loc == 0 or loc == -len(self)) and item + self.freq == self[0]:
   1526                     freq = self.freq
   1527                 elif (loc == len(self)) and item - self.freq == self[-1]:

c:\python\envs\pandas-0.16.1\lib\site-packages\pandas\tseries\index.pyc in __getitem__(self, key)
   1351         getitem = self._data.__getitem__
   1352         if np.isscalar(key):
-> 1353             val = getitem(key)
   1354             return Timestamp(val, offset=self.offset, tz=self.tz)
   1355         else:

IndexError: index 0 is out of bounds for axis 0 with size 0

```

Proposed https://github.com/pydata/pandas/pull/10194



Discussion:

This actually is a bit tricker. You are slicing another object, then trying to set it. This actually should hit the `SettingWIthCopyWarning` path, though it should actually work.

Side note is that we don't use `TimeSeries` anymore (its just a synonym for `Series` and will be deprecated in 0.17.0)

```
In [2]: ts2 = Series(0, pd.date_range('2011-01-01', '2011-01-01')) 

In [3]: ts2
Out[3]: 
2011-01-01    0
Freq: D, dtype: int64

In [4]: ts3 = ts2[:0]

In [5]: ts3
Out[5]: Series([], Freq: D, dtype: int64)

# this is the original structure
In [8]: ts3._data.blocks[0].values.base
Out[8]: array([0])
```

Actually, I've just found that setting an item on a newly created empty series with frequency still returns an error. Since it's a new series, there is no view involved. 

``` python
In [142]: pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.10.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 26 Stepping 5, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.16.2
nose: 1.3.4
Cython: 0.22
numpy: 1.9.2
scipy: 0.14.0
statsmodels: 0.6.1
IPython: 3.0.0
sphinx: 1.3
patsy: 0.3.0
dateutil: 2.4.2
pytz: 2015.4
bottleneck: 0.8.0
tables: 3.1.1
numexpr: 2.3.1
matplotlib: 1.4.3
openpyxl: 1.8.5
xlrd: 0.9.3
xlwt: 0.7.5
xlsxwriter: 0.6.7
lxml: 3.4.2
bs4: 4.3.2
html5lib: 0.999
httplib2: None
apiclient: None
sqlalchemy: 0.9.9
pymysql: None
psycopg2: None

In [143]: series = pd.Series([], freq='D')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-143-123100163557> in <module>()
----> 1 series = pd.Series([], freq='D')

TypeError: __init__() got an unexpected keyword argument 'freq'

In [144]: series = pd.Series([], pd.DatetimeIndex([], freq='D'))

In [145]: dt = pd.Timestamp('2011-01-01')

In [146]: series.loc[dt] = 9
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-146-c577a9a95757> in <module>()
----> 1 series.loc[dt] = 9

C:\dev\bin\Anaconda\envs\pandas-0.16.2\lib\site-packages\pandas-0.16.2-py2.7-win-amd64.egg\pandas\core\indexing.pyc in __setitem__(self, key, value)
    113     def __setitem__(self, key, value):
    114         indexer = self._get_setitem_indexer(key)
--> 115         self._setitem_with_indexer(indexer, value)
    116
    117     def _has_valid_type(self, k, axis):

C:\dev\bin\Anaconda\envs\pandas-0.16.2\lib\site-packages\pandas-0.16.2-py2.7-win-amd64.egg\pandas\core\indexing.pyc in _setitem_with_indexer(self, indexer, value)
    281                 if self.ndim == 1:
    282                     index = self.obj.index
--> 283                     new_index = index.insert(len(index),indexer)
    284
    285                     # this preserves dtype of the value

C:\dev\bin\Anaconda\envs\pandas-0.16.2\lib\site-packages\pandas-0.16.2-py2.7-win-amd64.egg\pandas\tseries\index.pyc in insert(self, loc, item)
   1499             # check freq can be preserved on edge cases
   1500             if self.freq is not None:
-> 1501                 if (loc == 0 or loc == -len(self)) and item + self.freq == self[0]:
   1502                     freq = self.freq
   1503                 elif (loc == len(self)) and item - self.freq == self[-1]:

C:\dev\bin\Anaconda\envs\pandas-0.16.2\lib\site-packages\pandas-0.16.2-py2.7-win-amd64.egg\pandas\tseries\base.pyc in __getitem__(self, key)
     73         getitem = self._data.__getitem__
     74         if np.isscalar(key):
---> 75             val = getitem(key)
     76             return self._box_func(val)
     77         else:

IndexError: index 0 is out of bounds for axis 0 with size 0


In [148]: series_nofreq = pd.Series([], pd.DatetimeIndex([]))

In [149]: series_nofreq.loc[pd.Timestamp('2011-01-01')] = 47

In [150]:
```


Resolving patch:
diff --git a/pandas/tseries/timedeltas.py b/pandas/tseries/timedeltas.py
--- a/pandas/tseries/timedeltas.py
+++ b/pandas/tseries/timedeltas.py
@@ -34,22 +34,13 @@ def _convert_listlike(arg, box, unit):
         if isinstance(arg, (list,tuple)) or ((hasattr(arg,'__iter__') and not hasattr(arg,'dtype'))):
             arg = np.array(list(arg), dtype='O')
 
+        # these are shortcutable
         if is_timedelta64_dtype(arg):
             value = arg.astype('timedelta64[ns]')
         elif is_integer_dtype(arg):
-
-            # these are shortcutable
-            value = arg.astype('timedelta64[{0}]'.format(unit)).astype('timedelta64[ns]')
+            value = arg.astype('timedelta64[{0}]'.format(unit)).astype('timedelta64[ns]', copy=False)
         else:
-            try:
-                value = tslib.array_to_timedelta64(_ensure_object(arg), unit=unit, coerce=coerce)
-            except:
-
-                # try to process strings fast; may need to fallback
-                try:
-                    value = np.array([ _get_string_converter(r, unit=unit)() for r in arg ],dtype='m8[ns]')
-                except:
-                    value = np.array([ _coerce_scalar_to_timedelta_type(r, unit=unit, coerce=coerce) for r in arg ])
+            value = tslib.array_to_timedelta64(_ensure_object(arg), unit=unit, coerce=coerce)
             value = value.astype('timedelta64[ns]', copy=False)
 
         if box:
@@ -95,15 +86,6 @@ def _convert_listlike(arg, box, unit):
     'NS' : 'ns',
     'ns' : 'ns',
     }
-_unit_scale = {
-    'd' : 86400*1e9,
-    'h' : 3600*1e9,
-    'm' : 60*1e9,
-    's' : 1e9,
-    'ms' : 1e6,
-    'us' : 1e3,
-    'ns' : 1,
-    }
 
 def _validate_timedelta_unit(arg):
     """ provide validation / translation for timedelta short units """
@@ -114,150 +96,11 @@ def _validate_timedelta_unit(arg):
             return 'ns'
         raise ValueError("invalid timedelta unit {0} provided".format(arg))
 
-_short_search = re.compile(
-    "^\s*(?P<neg>-?)\s*(?P<value>\d*\.?\d*)\s*(?P<unit>d|s|ms|us|ns)?\s*$",re.IGNORECASE)
-_full_search = re.compile(
-    "^\s*(?P<neg>-?)\s*(?P<days>\d*?\.?\d*?)?\s*(days|d|day)?,?\s*\+?(?P<time>\d{1,2}:\d{2}:\d{2})?(?P<frac>\.\d+)?\s*$",re.IGNORECASE)
-_nat_search = re.compile(
-    "^\s*(nat|nan)\s*$",re.IGNORECASE)
-_whitespace = re.compile('^\s*$')
-_number_split = re.compile("^(\d+\.?\d*)")
-
-# construct the full2_search
-abbrevs = [('d' ,'days|d|day'),
-           ('h' ,'hours|h|hour'),
-           ('m' ,'minutes|min|minute|m'),
-           ('s' ,'seconds|sec|second|s'),
-           ('ms','milliseconds|milli|millis|millisecond|ms'),
-           ('us','microseconds|micro|micros|microsecond|us'),
-           ('ns','nanoseconds|nano|nanos|nanosecond|ns')]
-
-_full_search2 = re.compile(''.join(
-    ["^\s*(?P<neg>-?)\s*"] + [ "(?P<" + p + ">\\d+\.?\d*\s*(" + ss + "))?\\s*" for p, ss in abbrevs ] + ['$']))
-
 def _coerce_scalar_to_timedelta_type(r, unit='ns', box=True, coerce=False):
     """ convert strings to timedelta; coerce to Timedelta (if box), else np.timedelta64"""
 
-    if isinstance(r, compat.string_types):
-
-        # we are already converting to nanoseconds
-        converter = _get_string_converter(r, unit=unit)
-        r = converter()
-        unit='ns'
-
     result = tslib.convert_to_timedelta(r,unit,coerce)
     if box:
         result = tslib.Timedelta(result)
 
     return result
-
-def _get_string_converter(r, unit='ns'):
-    """ return a string converter for r to process the timedelta format """
-
-    # treat as a nan
-    if isnull(r):
-        def convert(r=None, unit=None):
-            return tslib.iNaT
-        return convert
-
-    if _whitespace.search(r):
-        def convert(r=None, unit=None):
-            return tslib.iNaT
-        return convert
-
-    m = _short_search.search(r)
-    if m:
-        def convert(r=None, unit=unit, m=m):
-            if r is not None:
-                m = _short_search.search(r)
-
-            gd = m.groupdict()
-
-            r = float(gd['value'])
-            u = gd.get('unit')
-            if u is not None:
-                unit = u.lower()
-            result = tslib.cast_from_unit(r, unit)
-            if gd['neg']:
-                result *= -1
-            return result
-        return convert
-
-    m = _full_search.search(r)
-    if m:
-        def convert(r=None, unit=None, m=m):
-            if r is not None:
-                m = _full_search.search(r)
-
-            gd = m.groupdict()
-
-            # handle time
-            value = 0
-            time = gd['time']
-            if time:
-                (hh,mm,ss) = time.split(':')
-                value += int((float(hh)*3600 + float(mm)*60 + float(ss))*1e9)
-
-            # handle frac
-            frac = gd['frac']
-            if frac:
-                value += round(float(frac)*1e9)
-
-            # handle days (possibly negative)
-            is_neg = gd['neg']
-            if gd['days']:
-                days = int((float(gd['days'] or 0) * 86400)*1e9)
-                if is_neg:
-                    days *= -1
-                value += days
-            else:
-                if is_neg:
-                    value *= -1
-            return tslib.cast_from_unit(value, 'ns')
-        return convert
-
-    # look for combo strings
-    m = _full_search2.search(r)
-    if m:
-        def convert(r=None, unit=None, m=m):
-            if r is not None:
-                m = _full_search2.search(r)
-
-            gd = m.groupdict()
-
-            # the parser
-            def parse(k, v):
-                if v is None:
-                    return 0
-                v = float(_number_split.search(v).group())
-                return int(v*_unit_scale[k])
-
-            # handle non-days
-            days = gd.pop('days',None)
-            neg = gd.pop('neg',None)
-            value = 0
-            for k, v in gd.items():
-                value += parse(k,v)
-
-            # parse days / neg
-            if days:
-                days = parse('days',days)
-                if neg:
-                    days *= -1
-                value += days
-            else:
-                if neg:
-                    value *= -1
-
-            return tslib.cast_from_unit(value, 'ns')
-        return convert
-
-    m = _nat_search.search(r)
-    if m:
-        def convert(r=None, unit=None, m=m):
-            return tslib.iNaT
-        return convert
-
-    # no converter
-    raise ValueError("cannot create timedelta string converter for [{0}]".format(r))
-
diff --git a/pandas/tseries/tools.py b/pandas/tseries/tools.py
--- a/pandas/tseries/tools.py
+++ b/pandas/tseries/tools.py
@@ -263,6 +263,7 @@ def _convert_listlike(arg, box, format):
         if isinstance(arg, (list,tuple)):
             arg = np.array(arg, dtype='O')
 
+        # these are shortcutable
         if com.is_datetime64_ns_dtype(arg):
             if box and not isinstance(arg, DatetimeIndex):
                 try:
@@ -271,6 +272,12 @@ def _convert_listlike(arg, box, format):
                     pass
 
             return arg
+        elif format is None and com.is_integer_dtype(arg) and unit=='ns':
+            result = arg.astype('datetime64[ns]')
+            if box:
+                return DatetimeIndex(result, tz='utc' if utc else None)
+
+            return result
 
         arg = com._ensure_object(arg)
 


Test patch:
diff --git a/pandas/tseries/tests/test_timedeltas.py b/pandas/tseries/tests/test_timedeltas.py
--- a/pandas/tseries/tests/test_timedeltas.py
+++ b/pandas/tseries/tests/test_timedeltas.py
@@ -109,6 +109,9 @@ def test_construction(self):
         # currently invalid as it has a - on the hhmmdd part (only allowed on the days)
         self.assertRaises(ValueError, lambda : Timedelta('-10 days -1 h 1.5m 1s 3us'))
 
+        # only leading neg signs are allowed
+        self.assertRaises(ValueError, lambda : Timedelta('10 days -1 h 1.5m 1s 3us'))
+
         # roundtripping both for string and value
         for v in ['1s',
                   '-1s',
@@ -151,7 +154,7 @@ def test_construction(self):
                               "cannot construct a TimeDelta",
                               lambda : Timedelta())
         tm.assertRaisesRegexp(ValueError,
-                              "cannot create timedelta string convert",
+                              "unit abbreviation w/o a number",
                               lambda : Timedelta('foo'))
         tm.assertRaisesRegexp(ValueError,
                               "cannot construct a TimeDelta from the passed arguments, allowed keywords are ",