ENH: Allow unicode separators in to_csv & read_csv
State: closed

- single byte UTF8 separators will always work if utf8 encoding is specified (i.e. seps in range(256))
- single byte delimiters with encodings will always work in Python 3, but will not work in Python 2 if they are multibyte when coerced to UTF8
- multi byte delimiters with encodings will not work on reads with C engine and will not work with writes in Python 2. (but I believe they'll work in Python 3 if they're considered to be length 1).

Closes #6035.

This was slightly complicated validation-wise, but I think it's in a good place
with tests and such. I'll squash together the commits and make sure I've
covered the relevant cases in tests.

I also noticed some weirdness with passing `delimiter` as a keyword argument that
might need to be addressed in the future.

CC @maxgrenderjones, @Midnighter if you want to take a look.


Labels:
name: Bug
name: Unicode
description: Unicode strings
name: IO CSV
description: read_csv, to_csv

Comments:
to be clear, unicode requirement is because C engine expects delimiter to fit within a char. Wasn't quite sure if 128 bits is the max for a char (or if that's system-dependent)

obv not 128 bits, but ascii (i.e. max of 128 for ordinal)

I'll give it a try later tonight.

`UCHAR_MAX` is at least 255 according to C standard.

I wonder if that check is at all necessary: once the separator is encoded using requested encoding, it's a bytearray and each its element should fit in a char, hence `len(sep) == 1` check should be enough.

Also, accented latin characters would probably make a good test case: they are 1-byte in `latin1` encoding, but two bytes in `utf-8`:

``` python
In [4]: 'á'.encode('latin1')
Out[4]: b'\xe1'

In [5]: 'á'.encode('utf-8')
Out[5]: b'\xc3\xa1'
```

Or probably a broken bar which looks more like an actual separator:

``` python
In [8]: '¦'.encode('utf-8')
Out[8]: b'\xc2\xa6'

In [9]: '¦'.encode('latin1')
Out[9]: b'\xa6'

```

So tested the current changes and in Python 2.7 with `from __future__ import unicode_literals` this now works with any of the common separators. I then tried the following based on @immerrr 's comment:

``` python
for s in range(128, 256):
        df.to_csv("/tmp/out.csv", header=True, index=False, sep=unichr(s))
```

which fails at the moment - as expected. Traceback:

<pre>
Traceback (most recent call last):
  File "test.py", line 16, in <module>
    sep=unichr(s))
  File "/home/moritz/.virtualenvs/pd-uni/lib/python2.7/site-packages/pandas-0.11.0_5147_gd3b93f7-py2.7-linux-x86_64.egg/pandas/util/decorators.py", line 88, in wrapper
    return func(*args, **kwargs)
  File "/home/moritz/.virtualenvs/pd-uni/lib/python2.7/site-packages/pandas-0.11.0_5147_gd3b93f7-py2.7-linux-x86_64.egg/pandas/core/frame.py", line 1175, in to_csv
    escapechar=escapechar)
  File "/home/moritz/.virtualenvs/pd-uni/lib/python2.7/site-packages/pandas-0.11.0_5147_gd3b93f7-py2.7-linux-x86_64.egg/pandas/core/format.py", line 1181, in __init__
    sep = compat.binary_type(sep)
UnicodeEncodeError: 'ascii' codec can't encode character u'\x80' in position 0: ordinal not in range(128)
</pre>

Thanks for the comments folks. @Midnighter that's generating the wrong error message too. @immerrr - good point, coercing to bytes and checking length is a better way of doing it.

@Midnighter that's still going to fail now (just with a ValueError instead), since it can't figure out an encoding.

``` python
>>> df.to_csv('somepath.csv', unichr(256))
Traceback
    ...
ValueError: must specify single-byte separator compatible with encoding. ('ascii' codec can't encode character u'\u0100' in position 0: ordinal not in range(128))
```

hmm...it looks like it's a bit more complicated to support something like this in Python 2:

``` python
df.to_csv('out.csv', sep='¦', encoding='latin1')
```

because we have an intermediary step where we convert to utf8, write to the buffer and then recode to the specified output encoding. I think we have to mandate that (for writing in Python 2) it must be a 1-byte separator in utf8.

Okay, so because of issues with unicode and the CSV writer in Python 2, the separator has to be a single byte in UTF8 _in Python 2 only_. If someone has a strong use case for a non-UTF8 separator, we can consider that, but it's out of scope for this PR.  I've changed the exception type to be `NotImplementedError` in that case, to make that clearer.

So here's the overall behavior (and I've updated above as well):
- single byte UTF8 separators will always work if utf8 encoding is specified (i.e. seps in range(256))
- single byte delimiters with encodings will always work in Python 3, but will not work in Python 2 if they are multibyte when coerced to UTF8
- multi byte delimiters with encodings will not work on reads with C engine and will not work with writes in Python 2. (but I believe they'll work in Python 3 if they're considered to be length 1).

The tests now reflect the majority of this. I still need to add a source file that's a non-UTF8 encoding to see what the interaction is with all of this.

Just wanted to say thank you for putting so much work into what could be considered an arcane fringe case. It's much appreciated!

@Midnighter happy to do it :) one of those things where I started with 'oh it's easy to coerce `u(',')` to `','`' and then I just had to round out an unspec'd part of the library 

@jtratner can you rebase to master on this and we'll see the status?

closing pls reopen if/when updated