ENH: add arrow engine to read_csv
State: closed

- [x] closes #23697 
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

ASV's, 100,000 rows, 5 columns, for reading from BytesIO & StringIO buffers (running on 2 core machine).
```
[ 75.00%] ··· io.csv.ReadCSVEngine.time_read_bytescsv                                                                                                                                                                               ok
[ 75.00%] ··· ========= ============
                engine
              --------- ------------
                  c       51.7±6ms
                python    598±40ms
               pyarrow   27.8±0.8ms
              ========= ============

[100.00%] ··· io.csv.ReadCSVEngine.time_read_stringcsv                                                                                                                                                                              ok
[100.00%] ··· ========= ===========
                engine
              --------- -----------
                  c      53.8±20ms
                python    584±30ms
               pyarrow    29.9±1ms
              ========= ===========
```


Labels:
name: Enhancement
name: Performance
description: Memory or execution speed performance
name: IO CSV
description: read_csv, to_csv
name: Stale

Comments:
it appears that the arrow engine does better on larger datasets. for example, on scikit-learn's diabetes dataset(24 kb vs the 3kb iris), performance was ...
4.59 ms for arrow
6.83 ms for c
8.72 ms for python
@jorisvandenbossche @jreback, 
Is it okay to bump the pyarrow dependency up to 0.15 from 0.13 in order to support the skip_rows and column_names kwarg? 
i think would be ok to bump pyarrow min yes
can u d that as a separate change though 
> Is it okay to bump the pyarrow dependency up to 0.15 from 0.13 in order to support the skip_rows kwarg?

I would prefer to not bump the pyarrow version just for this (0.15 is really recent). I would maybe just check the pyarrow version and then give an error message that skip_rows is not supported if the version is too old?
@jorisvandenbossche Can you reconsider bumping the pyarrow dependency up to 0.15? Pyarrow 0.13 seems to miss a lot more keywords in addition to skiprows like include_columns(usecols), true_values, false_values, and also seems to only be able to use the first row as the column names. 
@lithomas1 we can certainly require 0.15 for this particular feature just not generally

Hello @lithomas1! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:













































There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-11-17 20:12:04 UTC
asv's, 100,000 rows, 5 columns
```
[ 75.00%] ··· io.csv.ReadCSVEngine.time_read_bytescsv                                                                                                                                                                               ok
[ 75.00%] ··· ========= ============
                engine
              --------- ------------
                  c       51.7±6ms
                python    598±40ms
               pyarrow   27.8±0.8ms
              ========= ============

[100.00%] ··· io.csv.ReadCSVEngine.time_read_stringcsv                                                                                                                                                                              ok
[100.00%] ··· ========= ===========
                engine
              --------- -----------
                  c      53.8±20ms
                python    584±30ms
               pyarrow    29.9±1ms
              ========= ===========
```
Would it make sense to support [`fletcher`](https://github.com/xhochy/fletcher/) `ExtensionArray` return types?

***c.c*** @xhochy 
> Would it make sense to support [`fletcher`](https://github.com/xhochy/fletcher/) `ExtensionArray` return types?
> 
> 
> 
> ***c.c*** @xhochy 

this is orthogonal
> > Would it make sense to support [`fletcher`](https://github.com/xhochy/fletcher/) `ExtensionArray` return types?
> > _**c.c**_ @xhochy
> 
> this is orthogonal

Not fully, as commented in https://github.com/pandas-dev/pandas/pull/31817/files#r440274357 you can easily extend this to read directly into ExtensionArray instances of all kind, not just `fletcher` but also basic pandas types like `IntegerArray`.
@lithomas1 it seems that in one of your last commits, you added a bunch of new code related to datetime conversions? Can you give a bit of explanation / context of what has been added.

I would also try to limit what is added here, and rather focus on getting basic csv reading merged. And then we can do follow-up PRs to add extra features (like support for extension arrays, better datetime parsing support with given formats, etc). 
Every time more features are added in this branch, that will require again new review rounds, and will only make it even longer to get this in. While smaller follow-up PRs will be easier to review.
@jorisvandenbossche 
At this point, I will not be adding any more features to the pyarrow engine. The date parsing capabilities were added because I feel like they are core csv features and because they would fail a lot of tests otherwise. The code for the date parsing capabalities is the same as the c/python parser code modified to handle dataframes(converting a dataframe to a col dict and back is too expensive). 

At this point, I just need to xfail all the failing tests to get CI to pass and adding all the xfails will probably increase the diffs by around 100 lines unfortunately. 
@jreback Can we reach a consensus on the fixtures so I can start xfailing the rest of the tests? 
> Can we reach a consensus on the fixtures so I can start xfailing the rest of the tests?

For the current usage of fixtures in the diff looks fine (I assume an alternative to the additional `pyarrow_xfail` fixture would be to have versions of the `all_parsers` fixture with and withou the pyarrow engine?) 
@jreback can you answer here?
will look soon

this needs to be passing though 
> this needs to be passing though

Yes, but @lithomas1 is checking if the approach is fine *before* doing the work to have all tests passing
put up what you want and will have a look
it's very hard to grok the way you are using the fixtures
@lithomas1 can you move release note to 1.2
@lithomas1 Can you merge master to fix the conflict?
I would like to see this move forward, and can also put some effort in updating it (in case @lithomas1 has no time on the short term).

@jreback can you clarify what exactly you don't like about the current `pyarrow_xfail` approach?
this needs a merge in master and a green build

will have a look
Yes it needs to be updated with latest master, but in the mean time it would still be useful to understand your concern about the fixture (as I asked in https://github.com/pandas-dev/pandas/pull/31817#issuecomment-654263399 already). So can you clarify your "it's very hard to grok the way you are using the fixtures" (see also @lithomas1's answers to your comments at https://github.com/pandas-dev/pandas/pull/31817#discussion_r441160898)
@jreback thanks a lot for the review! I added some comments/questions for further clarifications
Hi all, 
Sorry that I abandoned this PR for a while, and thanks to @simonjayhawkins  and @jorisvandenbossche for picking this up, there just seems to be way too many test cases that are failing than I can handle, last time I checked it was a couple hundred. Right now, there is a couple logic errors in the code with usecols I think, but code should be good otherwise. 

I still don't think I have the time/energy to finish this one though, but I can commit the WIP code that I have and provide some guidance on the code if necessary.
Continued in #38370