BUG, DOC: Allow custom line terminator with delim_whitespace=True
State: closed

Title is self-explanatory.  Closes #12912.


Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
The tokenizers in `tokenizer.c`, and the `CParser` tests in `test_parser.py` seem to have a lot of repetition, although reworking the latter seems easier to do than the former.   Can / should this PR also include refactoring changes to those parts of the code?

no make smaller / simpler PRs that do 1 thing

they can of course be built upon one another

why do you think adding all of this untested code is a good idea?

you would basically have to duplicate all of the test suite with a custom terminator in order to validate this

you need to use functions that are already tested
iow each of the cases needs to all a function 

What functions are you talking about?  If you look at the other tokenizer functions, they all duplicate the same or have very similar case work.  The function I added is in fact a near duplicate of `tokenize_whitespace` save replacing checks for `\n` and `\r` with `self->lineterminator`.  That's why I asked about the refactoring initially.

well then refactor first - we can't add more code like this

Managed to refactor code so that it only splits based on whether or not whitespace is used as a delimiter, so there are now only two tokenizing functions instead of three (or four after my second commit).  Will see if it can be squashed into just one, though it seems less straightforward compared to the custom line terminator split.

@gfyoung ok much better. pls also run asv's for csv to make sure no degredation.

So macros are good because they make the code shorter / more understandable while hopefully not sacrificing perf. more combining code is better (again could be done later), but since you are already working on it....

@jreback : there seems to have been a recent update to `python-dateutil` (just today), causing a test failure in `pandas.tslib` on OSX.  I can reproduce this failure on Linux FYI.

@gfyoung ok! my pip install catches things then, great! (the osx build will pip install python-dateutil to catch things just like this :>), while the 3.5 build uses conda (which is with a stable build). IIRC this has some reverses from previous things, so will have to sort them and see. Maybe we should ban certain versions of dateutil. These can cause weird things to happen.

@jreback : is it possible to put in a stopgap measure FTTB (e.g. change the `compat` condition in that test to being exactly `2.5.2` so that investigation can proceed in isolation from other PR's?)

see #12944 (I skipped the test for now on master), so rebase.

Whoot!  Squashed it down into one function!  Hopefully Travis will give the green light this time around.  I will also do some timing analysis for `read_csv` and post it in the conversation.

# For PR

<pre>io_bench.frame_to_csv.time_frame_to_csv           188.25ms
io_bench.frame_to_csv2.time_frame_to_csv2         312.94ms
...formatting.time_frame_to_csv_date_formatting    13.36ms
...h.frame_to_csv_mixed.time_frame_to_csv_mixed   218.47ms
...m.time_read_csv_infer_datetime_format_custom    15.82ms
....time_read_csv_infer_datetime_format_iso8601     2.64ms
..._ymd.time_read_csv_infer_datetime_format_ymd     2.92ms
...nch.read_csv_skiprows.time_read_csv_skiprows    16.74ms
...nch.read_csv_standard.time_read_csv_standard    14.00ms
..._dates_iso8601.time_read_parse_dates_iso8601     1.93ms
...h.write_csv_standard.time_write_csv_standard    50.49ms</pre>

# For Master

<pre>io_bench.frame_to_csv.time_frame_to_csv           195.29ms
io_bench.frame_to_csv2.time_frame_to_csv2         303.08ms
...formatting.time_frame_to_csv_date_formatting    12.99ms
...h.frame_to_csv_mixed.time_frame_to_csv_mixed   224.39ms
...m.time_read_csv_infer_datetime_format_custom    15.53ms
....time_read_csv_infer_datetime_format_iso8601     2.64ms
..._ymd.time_read_csv_infer_datetime_format_ymd     2.80ms
...nch.read_csv_skiprows.time_read_csv_skiprows    16.04ms
...nch.read_csv_standard.time_read_csv_standard    13.23ms
..._dates_iso8601.time_read_parse_dates_iso8601     1.78ms
...h.write_csv_standard.time_write_csv_standard    51.60ms</pre>

@jreback : Travis is giving the green light, and I don't see any major timing discrepancies.  Ready to merge if there is nothing else.

thanks @gfyoung didn't realize how much duplicative code there was already in tokenizer.c sheesh 👍 

as an aside, adding `delim_whitespace` option to python parser is trivial.