PERF: Improve perf of to_datetime with ISO format
State: closed

Closes #10178
Closes #8154

Using the example data from the issue:

```
In [2]: df = DataFrame({'date_text':["2015-05-18" for i in range(10**6)]})
```

Before:

```
In [3]: %timeit pd.to_datetime(df['date_text'],format="%Y-%m-%d", box=False).values.view('i8')/10**9
1 loops, best of 3: 3.14 s per loop

In [4]: %timeit pd.to_datetime(df['date_text'],infer_datetime_format=True,     box=False).values.view('i8')/10**9
1 loops, best of 3: 253 ms per lo
```

After:

```
In [6]: %timeit pd.to_datetime(df['date_text'],format="%Y-%m-%d", box=False).values.view('i8')/10**9
1 loops, best of 3: 217 ms per loop

In [7]: %timeit pd.to_datetime(df['date_text'],infer_datetime_format=True, box=False).values.view('i8')/10**9
1 loops, best of 3: 243 ms per loop
```


Labels:
name: Datetime
description: Datetime data dtype
name: Performance
description: Memory or execution speed performance

Comments:
is their an existing vbench for this? if not pls add one (but make 1/10 the length)

I added a vbench based on existing one that inferred the iso860 format.

```
---------------------------------------------------------
Test name                                    |    #0    |
---------------------------------------------------------
timeseries_to_datetime_iso8601_format        |   4.9593 |
timeseries_to_datetime_YYYYMMDD              |  10.0046 |
timeseries_to_datetime_iso8601               |   4.9581 |
---------------------------------------------------------
Test name                                    |    #0    |
---------------------------------------------------------
```

The second line of this test is failing - is that really the desired behavior?  The comments reference #10154  - Wasn't the conclusion that `errors='ignore'` should always suppress?

```
def test_day_not_in_month_coerce_false_ignore(self):
    self.assertEqual(to_datetime('2015-02-29', errors='ignore', coerce=False), '2015-02-29')
    self.assertRaises(ValueError, to_datetime, '2015-02-29', errors='ignore', format="%Y-%m-%d", coerce=False)
    self.assertRaises(ValueError, to_datetime, '2015-02-32', errors='ignore', format="%Y-%m-%d", coerce=False)
    self.assertRaises(ValueError, to_datetime, '2015-04-31', errors='ignore', format="%Y-%m-%d", coerce=False)
```

The intention of that test behavior is that yes you will ignore errors (and you are not coercing), BUT, since you specified the format it is still an invalid date. 

so you bring up the case that it is doing different things depending on whether a format is passed or not.

rereading those comments in #10154. I think it is just a choice. Ok, so let's make them the same. I'll reverse myself and now say that all 4 of these should raise. @chris-b1 would you make that change.

cc @jorisvandenbossche 
cc @vincentdavis

%Y format with integer data doesn't work on iso fast code (causing Stata test problems) so I've excluded from this and added a new test.

I also changed the error handling in `to_datetime` to always raise on out of bound / unparse-able date unless `coerce=True`.  It ends up breaking quite a few tests that expected the old behavior, just wanted to double check that's desired?  Thanks.

Rereading the comments in the other issue: https://github.com/pydata/pandas/issues/10154#issuecomment-106484971 and the two comments below. Based on that, I would say we agreed that with `errors='ignore'`, no error should be raised (although it is not really clear what we decided). But I did not see that the actual PR then did something else. 

I think it is pretty clear: if you give `errors='ignore'`, it should not raise an error. But as I said over there, I don't really like this default behaviour.

I think that makes sense.   One corner case that comes up - what should happen if the data passed doesn't match the format, but can all be parsed.  E.g.

```
pd.to_datetime(['2/28/2014', '2014-01-01'], format='%m/%d/%Y')
pd.to_datetime(['2/28/2014', '2014-01-01'], format='%m-%d-%Y')
```

I'm not sure, but it's probably least surprising if it fails? That's what the latest commit does - adding an additional parameter to `tslib.array_to_datetime` to only parse iso8601 strings.  I wasn't too sure on the cleanest way to do that, but here's a first attempt that at least seems to work.

@chris-b1 there are already a lot of paths thru this code, so getting very hard to grok. Any simplication would be appreciated.

After second thought, I am not sure anymore if this is a good idea (ignoring instead of raising), sorry ..

In any case, this is a backwards incompatible change, as it has been like that (that format raises instead of ignoring by default when it does not follow the format). It was just not correctly stated in the docstring. 
And if we do such a change, I would rather do it the other way around (change default of ignore to raise). But that is another discussion, and we should see if this change is not too breaking for people.

But maybe we can open a new issue for the discussion about the default behaviour, and keep this PR only for the perf improvement?

I think `_string_to_dts` should handle `require_iso8601` kw for simplification. Raise `DateParseError` when it is `True`, and `ValueError` otherwise. `array_to_datetime`, can pass through `DateParseError`, but catch `ValueError`. 
- https://github.com/pydata/pandas/blob/master/pandas/src/datetime.pxd#L127

For `errors` kw, I don't have clear opinion for now.

@sinhrks is it possible to raise/define an custom exception from a .pxd?  Sorry, running into the edge of my Cython knowledge.

@chris-b1 Can use cdef (and convert it to normal python object when catch). But it does not simplify as I first thought... Please forget, and I'll do separately if I find better idea.

@chris-b1 this looks ok. pls squash; ping when green.

I opened #10636 to address the changing of the default, which is prob a good idea for 0.17.0

@chris-b1 if you'd like to tackle would be gr8!

@jreback - should be good to merge.

@chris-b1 thanks!

on #10636 we'll have to put the new behavior (that was slightly changed here).