API/BUG: Make consistent datetime string parse function
State: closed

### Original Description

Allows `DatetimeIndex.__init__` to accept `NaT` string representation.
### Updated Description

The original problem is caused by the inconsistencies in parsing function. Make following results to be consistent.

```
date_str = '2011Q3'
pd.tseries.tools.parse_time_string(date_str)[0]
#2011-07-01 00:00:00

pd.DatetimeIndex([date_str])[0]
#2011-07-01 00:00:00

pd.tseries.tools.to_datetime(date_str)
# 2011Q3 <type str>, not parsed

pd.Timestamp(date_str)
# ValueError: Unable to parse 2011Q3
```

Summary: 
- Added `yearfirst` to `to_datetime`.
  - Defined private func `_to_datetime`, because `DatetimeIndex` uses `freq` kw to parse 6 character monthly strings (and do not want `to_datetime` to accept `freq`)
- `Timestamp` is left as it is (without `dayfirst` and `yearfirst`)

Also, closes #5418.
closes #10537 


Labels:
name: Datetime
description: Datetime data dtype
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
I think you might be able to completely eliminate `tseries/index/_str_to_dt_array` and instead just pass directly to `to_datetime` (you can leave this change though)

Currently removing `_str_to_dt_array` looks difficult, because it uses `parse_time_string` rather than `to_datetime`, and these functions have different logic / arguments. 

I think give it a try. the perf diff is huge.

@sinhrks did you look at perf?

pushing, if you get 2 this in next day or 2 lmk

@sinhrks revist?

 Sorry to delay. Removing `_str_to_dt_array` seems to be possible, but parsed result may be different because of inconsistencies in datetime parsing function.

Going to merge parsing function spec to solve this (changed the title so). 

Fixed `to_datetime` and `Timestamp` docstring to have `dayfirst` and `yearfirst` descriptions. 

And last difference I noticed is the case input only has time string. Currently, this PR doesn't change the  current master's behavior below.

```
pd.Timestamp("16:15")
# 2014-08-16 16:15:00

pd.tseries.tools.to_datetime("16:15")
# 2014-08-16 16:15:00

pd.tseries.tools.parse_time_string("16:15")[0]
# 0001-01-01 16:15:00
```

ok this is fine
ping on green

API question: Is it needed that `Timestamp()` gets the `dayfirst` and `yearfirst` keywords? 
In the reasoning: keep `Timestamp` simple and if you want more flexible parsing, then there is `to_datetime` to use (+ if we add them, why not also other keywords of `to_datetime` such as `format`?).

Whichever, but I think `dayfirst` and `yearfirst` are more important than others as these are passed to dateutil. 

@sinhrks something else, did you do a performance check? (as this touches quite some cython code)

@jorisvandenbossche in response to your API issue.

this is a bit of speculation, but I think that `to_datetime` was supposed to be the general convert me to datetimes, e.g. takes in a array/list/scalar and bunch of options to provide a very general parsing. While `Timestamp` is a pretty simple object with limited parsing.

That said, as long as it doesn't become a performance issue their is really no reason to not have `Timestamp` parse more generally.

There is maybe no real reason not to have it, but is there a good reason to have it? 

For me it feels like duplicating functionality for which there is no real need. Also, dayfirst being not strict and things like that, I don't know if we should encourage it.

If you have a program that has to parse a bunch of strings, then you use `to_datetime`. In my mind, the main usecase of directly calling `Timestamp` is when you just need to instantiate a certain datetime, and for that it is easy to provide a string (compared to the `datetime.datetime` constructor ..). But you also have direct control over the string you type in such case, and then is even more work to type `dayfirst=True` instead of just typing the date in the 'standard' format.

To be clear, I am also not strongly opposed to adding this, I just don't really see the need. But if there are other usecases where this would solve problems, then OK.

@sinhrks pls rebase. What's left here / what are the issues from above?

@jorisvandenbossche I think we had this discussion with 
cc @rockg

@sinhrks don't add yearfirst/datefirst to Timestamp

(this needs a rebase anyhow)

@sinhrks status?

@sinhrks status?

Is this PR in limbo? I just hit another case where I think this change would have helped out. (similar to #6961 , if you read a CSV with only year-month data it seems to use current day-of-month when filling out the date)

yep - want to pick it up (iirc it just needs to be scaled back a bit) don't add args to Timestamp

I also added those args back to the doc string in another PR.  Whoever picks up this PR, please put a comment in the source code referencing the PRs saying we don't want to add dayfirst and yearfirst to the doc string.  This will save having the same discussion yet another time.

@sinhrks ok I think we need to resurrect this (and just be minimal about its changes), or close.

thoughts?

Sorry, I'll revisit this. I have to understand `parse_time_string` little more to finish this.

OK, I think it is ready for review once again. I've attached summary on the top.

looks really good

can u run a vbench ?

Sorry to late. I did some changes to improve perfs, but there are still some benches get slower. I'm looking into this one by one.

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
groupby_ngroups_10000_cumcount               |  55.7633 |  48.1217 |   1.1588 |
groupby_transform_multi_key1                 |  51.6430 |  40.7820 |   1.2663 |
read_table_multiple_date_baseline            | 101.0714 |  74.2683 |   1.3609 |
read_table_multiple_date                     | 214.8240 | 156.3216 |   1.3742 |
series_xs_mi_ix                              |   2.1993 |   1.2210 |   1.8012 |
-------------------------------------------------------------------------------
```

Really nice clean-up!

Added some comments, and also a question about the `freq` keyword. You now made a `_to_datetime` to have this keyword, but is there a reason to not have this keyword in `to_datetime` itself? 

Also something else: I don't think we should target 0.16.1 here, as there are some changes in parsing behaviour for some cases.

@sinhrks this does clean up a bunch of stuff. can you run a perf check just to make sure. Let's move to 0.17.0 and put in first thing.

I've did further simplification at my local, but still have a performance regression like vbench attached above. I'll once summarize the changes after fixing the perf.