RFC/DOC: Standardize the column index type
State: closed

In [ARROW-1976](https://issues.apache.org/jira/browse/ARROW-1976) a user hit
an issue with serialization of unicode column index values using Python 2.

This PR proproses to make all column names into whatever the type of
`six.text_type` is, therefore handling Unicode in as uniform a way as possible
given the terrible state of Unicode in Python 2.

cc @wesm @TomAugspurger @jorisvandenbossche

I considered doing this by adding another field to indicate the individual type of the name, but that adds quite a bit of technical debt and backwards incompatibility. Since `json.loads`/`json.dumps` can read unicode strings across Pythons, this seemed like a backwards compatible way to support unicode column names across files written in different versions of Python (and pyarrow).

Labels:

Comments:
@jorisvandenbossche Not quite, because in Python 2 you can't call `str` successfully on arbitrary Unicode strings:

```
s = u'あ'
str(s)  # raises UnicodeDecodeError on Python 2
```
That's the bug in the reading part? (I meant that now strings are already written as unicode to the parquet file, hence giving an error in python 2 on reading with str() ?)
What specifically is the proposal here, is the restriction on writing or reading? I think we should be restricting what is written.

And can we go further and require a specific encoding (utf-8)? Or is that too onerous for some applications?
@jorisvandenbossche 

No, you cannot convert them to `arrow.Table` objects:

```python
df = pd.DataFrame({u'あ': [u'い']}, index=[u'う'])
pa.Table.from_pandas(df)  # raises an error
```

because we're calling `str` on `field_name`
In `pa.Table.from_pandas` any unicode field names should be converted to UTF-8 bytes, where is the problem occurring?
In `pa.Table.from_pandas` any unicode field names should be converted to UTF-8 bytes, where is the problem occurring?
> What specifically is the proposal here, is the restriction on writing or reading? I think we should be restrict what is written.

@TomAugspurger 

> Column names will be serialized using whatever type ``six.text_type`` corresponds to.

This is a little ambiguous :) This proposal is to restrict writing.

@wesm 

1. https://github.com/apache/arrow/blob/master/python/pyarrow/pandas_compat.py#L288
2. https://github.com/apache/arrow/blob/master/python/pyarrow/pandas_compat.py#L175
3. Potentially here: https://github.com/apache/arrow/blob/master/python/pyarrow/pandas_compat.py#L236
Here as well: https://github.com/apache/arrow/blob/master/python/pyarrow/pandas_compat.py#L330

since that will skip over `unicode` instances
I'm on bad internet so I'll have to look at the code later. At a glance it sounds like we just have bugs in pandas_compat.py that need fixing, e.g. it's unclear why calling `str` on the field names is a requirement and why it's causing problems
@jorisvandenbossche @TomAugspurger @wesm 

@wesm is right, there are actually just bugs in `pandas_compat.py`
I'm going to close this PR.
> [about unicode column names] No, you cannot convert them to arrow.Table objects:

OK, was only trying on python 3, and there that works perfectly (at least with pyarrow 0.8)

I agree that it not working in python 2 were just bugs in arrow, but I think it is still worth including some language about this here (the fact that column names are converted to unicode on writing). Because it would be nice that other implementation write column names in the same way to the metadata.


+1. I would say "String column names should be UTF-8 encoded."

On Sat, Feb 3, 2018 at 11:02 AM, Joris Van den Bossche <
notifications@github.com> wrote:

> [about unicode column names] No, you cannot convert them to arrow.Table
> objects:
>
> OK, was only trying on python 3, and there that works perfectly (at least
> with pyarrow 0.8)
>
> I agree that it not working in python 2 were just bugs in arrow, but I
> think it is still worth including some language about this here (the fact
> that column names are converted to unicode on writing). Because it would be
> nice that other implementation write column names in the same way to the
> metadata.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/19519#issuecomment-362798143>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIlES6b1R4HAM8MIyV8518i0l1WxBks5tRDy3gaJpZM4R3y7e>
> .
>