Repository: pandas-dev/pandas

Issue:
Fastpath for to_datetime when providing ISO format as keyword?
Say you have to parse some nicely ISO formatted date strings, you can just parse this with `todatetime` very fast. But if you were 'overcautious' and provided the `format="%Y-%m-%d %H:%M:%S"` for safety, this seems to be around 20 times slower. 
Would it be possible to provide a fastpath for certain provided format strings (as already exists for `%Y%m%d` I think).

```
In [129]: s = pd.Series(pd.date_range('2000-01-01', periods=1000, freq='H'))

In [130]: s_as_dt_strings = s.apply(lambda x: x.strftime("%Y-%m-%dT%H:%M:%S.%f"))

In [131]: %timeit pd.to_datetime(s_as_dt_strings)
1000 loops, best of 3: 406 µs per loop

In [132]: %timeit pd.to_datetime(s_as_dt_strings, format="%Y-%m-%dT%H:%M:%S.%f")
100 loops, best of 3: 9.73 ms per loop
```

```
In [133]: s_as_dt_strings = s.apply(lambda x: x.strftime("%Y-%m-%d %H:%M:%S"))

In [134]: %timeit pd.to_datetime(s_as_dt_strings)
1000 loops, best of 3: 361 µs per loop

In [135]: %timeit pd.to_datetime(s_as_dt_strings, format="%Y-%m-%d %H:%M:%S")
100 loops, best of 3: 8.36 ms per loop
```

For non-standard formats, providing format _does_ give a big improvement:

```
In [136]: s_as_dt_strings = s.apply(lambda x: x.strftime("%Y/%m/%d %H:%M:%S"))

In [137]: %timeit pd.to_datetime(s_as_dt_strings)
10 loops, best of 3: 92.2 ms per loop

In [138]: %timeit pd.to_datetime(s_as_dt_strings, format="%Y/%m/%d %H:%M:%S")
100 loops, best of 3: 9.08 ms per loop
```



Discussion:
if u provide a format I think it bypasses the infer_datetime_format option which is much faster if it knows the format

so need to short circuit on certain formats

But the `infer_datetime_format` is False by default no?

```
In [134]: %timeit pd.to_datetime(s_as_dt_strings)
1000 loops, best of 3: 382 µs per loop

In [135]: %timeit pd.to_datetime(s_as_dt_strings, infer_datetime_format=True)
1000 loops, best of 3: 963 µs per loop

In [136]: %timeit pd.to_datetime(s_as_dt_strings, format="%Y-%m-%dT%H:%M:%S.%f")
100 loops, best of 3: 9.36 ms per loop
```

But indeed, the fact that it is faster with `infer_datetime_format` (which guesses the same format as I provide with `format=`, I would think) seems to indicate that with providing `format=` it could also be faster by using some same short circuiting

exactly

I think needs a simple check if format is provided if we can fastpath it (eg for recognized formats), if so then just follow the infer_datetime_format=True path and ignore the format); I think this is done for YYYYMMDD (only) atm

or in the basic format whee we don't need to infer at all just pass it thru (eg your first timeit with the 3rd format)

With `infer_datetime_format` there is indeed a bypass for iso formatted strings: https://github.com/pydata/pandas/blob/master/pandas/tseries/tools.py#L251, so this should maybe also be done with provided `formats`


Resolving patch:
diff --git a/pandas/tseries/tools.py b/pandas/tseries/tools.py
--- a/pandas/tseries/tools.py
+++ b/pandas/tseries/tools.py
@@ -296,21 +296,24 @@ def _convert_listlike(arg, box, format):
             return result
 
         arg = com._ensure_object(arg)
+        require_iso8601 = False
 
         if infer_datetime_format and format is None:
             format = _guess_datetime_format_for_array(arg, dayfirst=dayfirst)
 
-            if format is not None:
-                # There is a special fast-path for iso8601 formatted
-                # datetime strings, so in those cases don't use the inferred
-                # format because this path makes process slower in this
-                # special case
-                format_is_iso8601 = (
-                    '%Y-%m-%dT%H:%M:%S.%f'.startswith(format) or
-                    '%Y-%m-%d %H:%M:%S.%f'.startswith(format)
-                )
-                if format_is_iso8601:
-                    format = None
+        if format is not None:
+            # There is a special fast-path for iso8601 formatted
+            # datetime strings, so in those cases don't use the inferred
+            # format because this path makes process slower in this
+            # special case
+            format_is_iso8601 = (
+                ('%Y-%m-%dT%H:%M:%S.%f'.startswith(format) or
+                '%Y-%m-%d %H:%M:%S.%f'.startswith(format)) and
+				format != '%Y'
+            )
+            if format_is_iso8601:
+                require_iso8601 = not infer_datetime_format
+                format = None
 
         try:
             result = None
@@ -334,16 +337,20 @@ def _convert_listlike(arg, box, format):
                             raise
                         result = arg
                     except ValueError:
-                        # Only raise this error if the user provided the
-                        # datetime format, and not when it was inferred
+                        # if format was inferred, try falling back
+                        # to array_to_datetime - terminate here
+                        # for specified formats
                         if not infer_datetime_format:
-                            raise
+                            if errors == 'raise':
+                                raise
+                            result = arg
 
             if result is None and (format is None or infer_datetime_format):
-                result = tslib.array_to_datetime(arg, raise_=errors == 'raise',
+                result = tslib.array_to_datetime(arg, raise_=errors=='raise',
                                                  utc=utc, dayfirst=dayfirst,
                                                  yearfirst=yearfirst, freq=freq,
-                                                 coerce=coerce, unit=unit)
+                                                 coerce=coerce, unit=unit,
+                                                 require_iso8601=require_iso8601)
 
             if com.is_datetime64_dtype(result) and box:
                 result = DatetimeIndex(result, tz='utc' if utc else None)
diff --git a/vb_suite/timeseries.py b/vb_suite/timeseries.py
--- a/vb_suite/timeseries.py
+++ b/vb_suite/timeseries.py
@@ -157,6 +157,10 @@ def date_range(start=None, end=None, periods=None, freq=None):
     Benchmark('to_datetime(strings)', setup,
               start_date=datetime(2012, 7, 11))
 
+timeseries_to_datetime_iso8601_format = \
+    Benchmark("to_datetime(strings, format='%Y-%m-%d %H:%M:%S')", setup,
+              start_date=datetime(2012, 7, 11))
+
 setup = common_setup + """
 rng = date_range('1/1/2000', periods=10000, freq='D')
 strings = Series(rng.year*10000+rng.month*100+rng.day,dtype=np.int64).apply(str)


Test patch:
diff --git a/pandas/tseries/tests/test_timeseries.py b/pandas/tseries/tests/test_timeseries.py
--- a/pandas/tseries/tests/test_timeseries.py
+++ b/pandas/tseries/tests/test_timeseries.py
@@ -919,8 +919,8 @@ def test_to_datetime_with_apply(self):
         assert_series_equal(result, expected)
 
         td = pd.Series(['May 04', 'Jun 02', ''], index=[1,2,3])
-        self.assertRaises(ValueError, lambda : pd.to_datetime(td,format='%b %y'))
-        self.assertRaises(ValueError, lambda : td.apply(pd.to_datetime, format='%b %y'))
+        self.assertRaises(ValueError, lambda : pd.to_datetime(td,format='%b %y', errors='raise'))
+        self.assertRaises(ValueError, lambda : td.apply(pd.to_datetime, format='%b %y', errors='raise'))
         expected = pd.to_datetime(td, format='%b %y', coerce=True)
 
         result = td.apply(lambda x: pd.to_datetime(x, format='%b %y', coerce=True))
@@ -4197,6 +4197,20 @@ def test_to_datetime_format_YYYYMMDD(self):
         expected = Series(['20121231','20141231','NaT'],dtype='M8[ns]')
         assert_series_equal(result, expected)
 
+    # GH 10178
+    def test_to_datetime_format_integer(self):
+        s = Series([2000, 2001, 2002])
+        expected = Series([ Timestamp(x) for x in s.apply(str) ])
+
+        result = to_datetime(s,format='%Y')
+        assert_series_equal(result, expected)
+
+        s = Series([200001, 200105, 200206])
+        expected = Series([ Timestamp(x[:4] + '-' + x[4:]) for x in s.apply(str) ])
+
+        result = to_datetime(s,format='%Y%m')
+        assert_series_equal(result, expected)
+
     def test_to_datetime_format_microsecond(self):
         val = '01-Apr-2011 00:00:01.978'
         format = '%d-%b-%Y %H:%M:%S.%f'
@@ -4524,9 +4538,9 @@ def test_day_not_in_month_coerce_false_raise(self):
 
     def test_day_not_in_month_coerce_false_ignore(self):
         self.assertEqual(to_datetime('2015-02-29', errors='ignore', coerce=False), '2015-02-29')
-        self.assertRaises(ValueError, to_datetime, '2015-02-29', errors='ignore', format="%Y-%m-%d", coerce=False)
-        self.assertRaises(ValueError, to_datetime, '2015-02-32', errors='ignore', format="%Y-%m-%d", coerce=False)
-        self.assertRaises(ValueError, to_datetime, '2015-04-31', errors='ignore', format="%Y-%m-%d", coerce=False)
+        self.assertEqual(to_datetime('2015-02-29', errors='ignore', format="%Y-%m-%d", coerce=False), '2015-02-29')
+        self.assertEqual(to_datetime('2015-02-32', errors='ignore', format="%Y-%m-%d", coerce=False), '2015-02-32')
+        self.assertEqual(to_datetime('2015-04-31', errors='ignore', format="%Y-%m-%d", coerce=False), '2015-04-31')
 
 if __name__ == '__main__':
     nose.runmodule(argv=[__file__, '-vvs', '-x', '--pdb', '--pdb-failure'],