Repository: pandas-dev/pandas
Pull request: Remove Encoding of values in char** For Labels
- [X] closes #20500

In reviewing this module there is a shared function for object keys and values which encodes objects into a separate buffer and subsequently indexes off of that. Instead of encoding values in a buffer I've updated that function to be a char ** pointing to string representations of the labels (or index / columns, rather). 

This is arguably a pre-cursor to:

  1. #19486 to disentangle date time functions from JSON
  2. #12004 to add indent support (tried this previously but vendoring ujson updates but didn't work because of this limitation)
  3. #27164 because the various formats may need column / index labels at different points in time, so encoding up front makes that very difficult to reuse

The only downside here I haven't been able to figure out is how to deal with date formatting. Right now all labels are written as epochs. I'm sure there is a way to handle but I wasn't clear on what the best way to convert arbitrary input (i.e. object or datetime dtypes) into ISO formats by element where applicable.

cc @jbrockmendel in case you have insight on that

Problem statement:
DataFrame.to_json() produces malformed JSON when DataFrame contains tuples as column
#### Code Sample, a copy-pastable example if possible

```python
# Your code here
# coding: utf-8
import json, traceback, pandas
test = pandas.DataFrame(data = {
    'key': ['a', 'a', 'b', 'b', 'a'],
    'value': [1, 2, 3, 4, 5]
})
stat = test.groupby('key').agg(['sum', 'mean', 'count'])
stat_json = stat.to_json()
print(stat)
print(stat_json)
try:
    json.loads(stat_json)
except ValueError:
    traceback.print_exc()
stat_reset = stat.reset_index()
stat_reset_json = stat_reset.to_json()
print(stat_reset)
print(stat_reset_json)
try:
    json.loads(stat_reset_json)
except ValueError:
    traceback.print_exc()
```

##### Result by Python 2.7

```
    value                
      sum      mean count
key                      
a       8  2.666667     3
b       7  3.500000     2
{"["value","sum"]":{"a":8,"b":7},"["value","mean"]":{"a":2.6666666667,"b":3.5},"["value","count"]":{"a":3,"b":2}}
Traceback (most recent call last):
  File "pycheck.py", line 12, in <module>
    json.loads(stat_json)
  File "/usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 380, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Expecting : delimiter: line 1 column 5 (char 4)
  key value                
        sum      mean count
0   a     8  2.666667     3
1   b     7  3.500000     2
{"["key",""]":{"0":"a","1":"b"},"["value","sum"]":{"0":8,"1":7},"["value","mean"]":{"0":2.6666666667,"1":3.5},"["value","count"]":{"0":3,"1":2}}
Traceback (most recent call last):
  File "pycheck.py", line 20, in <module>
    json.loads(stat_reset_json)
  File "/usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 380, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Expecting : delimiter: line 1 column 5 (char 4)
```

##### Result by Python 3
```
    value                
      sum      mean count
key                      
a       8  2.666667     3
b       7  3.500000     2
{"["value","sum"]":{"a":8,"b":7},"["value","mean"]":{"a":2.6666666667,"b":3.5},"["value","count"]":{"a":3,"b":2}}
Traceback (most recent call last):
  File "pycheck.py", line 12, in <module>
    json.loads(stat_json)
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ':' delimiter: line 1 column 5 (char 4)
  key value                
        sum      mean count
0   a     8  2.666667     3
1   b     7  3.500000     2
{"["key",""]":{"0":"a","1":"b"},"["value","sum"]":{"0":8,"1":7},"["value","mean"]":{"0":2.6666666667,"1":3.5},"["value","count"]":{"0":3,"1":2}}
Traceback (most recent call last):
  File "pycheck.py", line 20, in <module>
    json.loads(stat_reset_json)
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ':' delimiter: line 1 column 5 (char 4)
```

#### Problem description

``DataFrame.to_json()`` returns malformed JSON when its column contains tuple object, such as ``('value', 'sum')``, ``('value', 'mean')`` etc. in this case.

#### Expected Output

I think, at least in this case, correct output of double quote in strings should be escaped like this:

```
{"[\"value\",\"sum\"]":{"a":8,"b":7},"[\"value\",\"mean\"]":{"a":2.6666666667,"b":3.5},"[\"value\",\"count\"]":{"a":3,"b":2}}

{"[\"key\",\"\"]":{"0":"a","1":"b"},"[\"value\",\"sum\"]":{"0":8,"1":7},"[\"value\",\"mean\"]":{"0":2.6666666667,"1":3.5},"[\"value\",\"count\"]":{"0":3,"1":2}}
```

#### Output of ``pd.show_versions()``

##### Python2.7

<details>

```
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.14.final.0
python-bits: 64
OS: Darwin
OS-release: 16.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.22.0
pytest: None
pip: 9.0.1
setuptools: 38.5.2
Cython: None
numpy: 1.14.1
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: 1.6.3
patsy: None
dateutil: 2.7.0
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.5.3
html5lib: None
sqlalchemy: 1.2.5
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
```

</details>

##### Python3

<details>

```
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Darwin
OS-release: 16.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.22.0
pytest: None
pip: 9.0.1
setuptools: 38.5.2
Cython: None
numpy: 1.14.1
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.0
pytz: 2018.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: 1.2.5
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
```

</details>

_I have checked past issues briefly, but I'm sorry if same issue was already filed._

Hints from issue discussion:
I'm not sure about the expected behavior here. How do you represent multiple-part keys in JSON?
I don't know what is the best solution that can satisfy everyone.

According to the [JSON specification](http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf), including value other than string as object's key is impossible.

In order to avoid this problem in my work, I usually eliminate all tuples and manage to get string expression in some way, before writing JSON:

```python
import json, pandas
def eliminate_column_tuple(df, glue):
    columns = df.columns
    new_columns = []
    for column in columns:
        if type(column) is tuple:
            joined = glue.join(column)
            if joined.endswith(glue):
                joined = joined[:-len(glue)]
            new_columns.append(joined)
        else:
            new_columns.append(column)
    df.columns = new_columns
    return df
test = pandas.DataFrame(data = {
    'key': ['a', 'a', 'b', 'b', 'a'],
    'value': [1, 2, 3, 4, 5]
})
stat = test.groupby('key').agg(['sum', 'mean', 'count'])
simple_stat = eliminate_column_tuple(stat, '|')
stat_json = simple_stat.to_json()
print(stat_json)
json.loads(stat_json)
```

However, such kind of solution strongly depends on the purpose.

If this problem won't be fixed, at least it is recommended to write into the API document that `to_json()` is useless for `DataFrame` with tuple columns.

I'd prefer to raise an exception (like Python does) rather than document
that we produce invalid JSON.

On Tue, Mar 27, 2018 at 8:37 PM, k-yaegashi <notifications@github.com>
wrote:

> I don't know what is the best solution that can satisfy everyone.
>
> According to the JSON specification
> <http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf>,
> including value other than string as object's key is impossible.
>
> In order to avoid this problem in my work, I usually eliminate all tuples
> and manage to get string expression in some way, before writing JSON:
>
> import json, pandasdef eliminate_column_tuple(df, glue):
>     columns = df.columns
>     new_columns = []
>     for column in columns:
>         if type(column) is tuple:
>             joined = glue.join(column)
>             if joined.endswith(glue):
>                 joined = joined[:-len(glue)]
>             new_columns.append(joined)
>         else:
>             new_columns.append(column)
>     df.columns = new_columns
>     return df
> test = pandas.DataFrame(data = {
>     'key': ['a', 'a', 'b', 'b', 'a'],
>     'value': [1, 2, 3, 4, 5]
> })
> stat = test.groupby('key').agg(['sum', 'mean', 'count'])
> simple_stat = eliminate_column_tuple(stat, '|')
> stat_json = simple_stat.to_json()print(stat_json)
> json.loads(stat_json)
>
> However, such kind of solution strongly depends on the purpose.
>
> If this problem won't be fixed, at least it is recommended to write into
> the API document that to_json() is useless for DataFrame with tuple
> columns.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/20500#issuecomment-376730925>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIpLHwxdZaj8caZxw0CVbnVHqwkDcks5tiulxgaJpZM4S81Ul>
> .
>

yeah I agree, can put a check here that if an index is ``object`` dtype then

```
In [6]: infer_dtype = pd.api.types.infer_dtype

In [8]: infer_dtype(pd.Index([1,2,3]))
Out[8]: 'integer'

In [9]: infer_dtype(pd.Index(['foo', 'bar', 'baz']))
Out[9]: 'string'

In [10]: infer_dtype(pd.Index(['foo', 'bar', 1]))
Out[10]: 'mixed-integer'

In [11]: infer_dtype(pd.Index(['foo', 'bar', (1, 2)]))
Out[11]: 'mixed'
```

prob allow [10] but not [11]

Commit message: whitespace cleanup

Patch to review:
diff --git a/pandas/_libs/src/ujson/python/objToJSON.c b/pandas/_libs/src/ujson/python/objToJSON.c
index 926440218b5d9..aafa0544d4b09 100644
--- a/pandas/_libs/src/ujson/python/objToJSON.c
+++ b/pandas/_libs/src/ujson/python/objToJSON.c
@@ -787,30 +787,23 @@ JSOBJ NpyArr_iterGetValue(JSOBJ obj, JSONTypeContext *tc) {
     return GET_TC(tc)->itemValue;
 }
 
-static void NpyArr_getLabel(JSOBJ obj, JSONTypeContext *tc, size_t *outLen,
-                            npy_intp idx, char **labels) {
-    JSONObjectEncoder *enc = (JSONObjectEncoder *)tc->encoder;
-    PRINTMARK();
-    *outLen = strlen(labels[idx]);
-    Buffer_Reserve(enc, *outLen);
-    memcpy(enc->offset, labels[idx], sizeof(char) * (*outLen));
-    enc->offset += *outLen;
-    *outLen = 0;
-}
-
 char *NpyArr_iterGetName(JSOBJ obj, JSONTypeContext *tc, size_t *outLen) {
     NpyArrContext *npyarr = GET_TC(tc)->npyarr;
     npy_intp idx;
     PRINTMARK();
+    char *cStr;
 
     if (GET_TC(tc)->iterNext == NpyArr_iterNextItem) {
         idx = npyarr->index[npyarr->stridedim] - 1;
-        NpyArr_getLabel(obj, tc, outLen, idx, npyarr->columnLabels);
+	cStr = npyarr->columnLabels[idx];
     } else {
         idx = npyarr->index[npyarr->stridedim - npyarr->inc] - 1;
-        NpyArr_getLabel(obj, tc, outLen, idx, npyarr->rowLabels);
+	cStr = npyarr->rowLabels[idx];
     }
-    return NULL;
+
+    *outLen = strlen(cStr);
+
+    return cStr;
 }
 
 //=============================================================================
@@ -852,19 +845,22 @@ char *PdBlock_iterGetName(JSOBJ obj, JSONTypeContext *tc, size_t *outLen) {
     PdBlockContext *blkCtxt = GET_TC(tc)->pdblock;
     NpyArrContext *npyarr = blkCtxt->npyCtxts[0];
     npy_intp idx;
+    char *cStr;
     PRINTMARK();
 
     if (GET_TC(tc)->iterNext == PdBlock_iterNextItem) {
         idx = blkCtxt->colIdx - 1;
-        NpyArr_getLabel(obj, tc, outLen, idx, npyarr->columnLabels);
+	cStr = npyarr->columnLabels[idx];
     } else {
         idx = GET_TC(tc)->iterNext != PdBlock_iterNext
                   ? npyarr->index[npyarr->stridedim - npyarr->inc] - 1
                   : npyarr->index[npyarr->stridedim];
 
-        NpyArr_getLabel(obj, tc, outLen, idx, npyarr->rowLabels);
+	cStr = npyarr->rowLabels[idx];
     }
-    return NULL;
+
+    *outLen = strlen(cStr);
+    return cStr;
 }
 
 char *PdBlock_iterGetName_Transpose(JSOBJ obj, JSONTypeContext *tc,
@@ -872,16 +868,19 @@ char *PdBlock_iterGetName_Transpose(JSOBJ obj, JSONTypeContext *tc,
     PdBlockContext *blkCtxt = GET_TC(tc)->pdblock;
     NpyArrContext *npyarr = blkCtxt->npyCtxts[blkCtxt->colIdx];
     npy_intp idx;
+    char *cStr;
     PRINTMARK();
 
     if (GET_TC(tc)->iterNext == NpyArr_iterNextItem) {
         idx = npyarr->index[npyarr->stridedim] - 1;
-        NpyArr_getLabel(obj, tc, outLen, idx, npyarr->columnLabels);
+	cStr = npyarr->columnLabels[idx];
     } else {
         idx = blkCtxt->colIdx;
-        NpyArr_getLabel(obj, tc, outLen, idx, npyarr->rowLabels);
+	cStr = npyarr->rowLabels[idx];
     }
-    return NULL;
+
+    *outLen = strlen(cStr);
+    return cStr;
 }
 
 int PdBlock_iterNext(JSOBJ obj, JSONTypeContext *tc) {
@@ -1578,15 +1577,31 @@ void NpyArr_freeLabels(char **labels, npy_intp len) {
     }
 }
 
+/*
+ * Function: NpyArr_encodeLabels
+ * -----------------------------
+ *
+ * Builds an array of "encoded" labels.
+ *
+ * labels: PyArrayObject pointer for labels to be "encoded"
+ * num : number of labels
+ *
+ * "encode" is quoted above because we aren't really doing encoding
+ * For historical reasons this function would actually encode the entire
+ * array into a separate buffer with a separate call to JSON_Encode
+ * and would leave it to complex pointer manipulation from there to
+ * unpack values as needed. To make things simpler and more idiomatic
+ * this has instead just stringified any input save for datetime values,
+ * which may need to be represented in various formats.
+ */
 char **NpyArr_encodeLabels(PyArrayObject *labels, JSONObjectEncoder *enc,
                            npy_intp num) {
     // NOTE this function steals a reference to labels.
     PyObjectEncoder *pyenc = (PyObjectEncoder *)enc;
     PyObject *item = NULL;
-    npy_intp i, stride, len, need_quotes;
+    npy_intp i, stride, len;
     char **ret;
-    char *dataptr, *cLabel, *origend, *origst, *origoffset;
-    char labelBuffer[NPY_JSON_BUFSIZE];
+    char *dataptr, *cLabel;
     PyArray_GetItemFunc *getitem;
     int type_num;
     PRINTMARK();
@@ -1614,68 +1629,44 @@ char **NpyArr_encodeLabels(PyArrayObject *labels, JSONObjectEncoder *enc,
         ret[i] = NULL;
     }
 
-    origst = enc->start;
-    origend = enc->end;
-    origoffset = enc->offset;
-
     stride = PyArray_STRIDE(labels, 0);
     dataptr = PyArray_DATA(labels);
     getitem = (PyArray_GetItemFunc *)PyArray_DESCR(labels)->f->getitem;
     type_num = PyArray_TYPE(labels);
 
     for (i = 0; i < num; i++) {
-        if (PyTypeNum_ISDATETIME(type_num) || PyTypeNum_ISNUMBER(type_num))
-        {
-            item = (PyObject *)labels;
-            pyenc->npyType = type_num;
-            pyenc->npyValue = dataptr;
-        } else {
-            item = getitem(dataptr, labels);
-            if (!item) {
-                NpyArr_freeLabels(ret, num);
-                ret = 0;
-                break;
-            }
-        }
-
-        cLabel = JSON_EncodeObject(item, enc, labelBuffer, NPY_JSON_BUFSIZE);
-
-        if (item != (PyObject *)labels) {
-            Py_DECREF(item);
-        }
-
-        if (PyErr_Occurred() || enc->errorMsg) {
+        item = getitem(dataptr, labels);
+        if (!item) {
+	  NpyArr_freeLabels(ret, num);
+	  ret = 0;
+	  break;
+	}
+
+	PyObject *str = PyObject_Str(item);
+	cLabel = PyUnicode_AsUTF8(str);
+	len = strlen(cLabel);
+	Py_DECREF(str);
+
+	Py_DECREF(item);
+	// Add 1 to include NULL terminator
+	ret[i] = PyObject_Malloc(len + 1);
+	memcpy(ret[i], cLabel, len + 1);
+
+        if (PyErr_Occurred()) {
             NpyArr_freeLabels(ret, num);
             ret = 0;
             break;
         }
 
-        need_quotes = ((*cLabel) != '"');
-        len = enc->offset - cLabel + 1 + 2 * need_quotes;
-        ret[i] = PyObject_Malloc(sizeof(char) * len);
-
         if (!ret[i]) {
             PyErr_NoMemory();
             ret = 0;
             break;
         }
 
-        if (need_quotes) {
-            ret[i][0] = '"';
-            memcpy(ret[i] + 1, cLabel, sizeof(char) * (len - 4));
-            ret[i][len - 3] = '"';
-        } else {
-            memcpy(ret[i], cLabel, sizeof(char) * (len - 2));
-        }
-        ret[i][len - 2] = ':';
-        ret[i][len - 1] = '\0';
         dataptr += stride;
     }
 
-    enc->start = origst;
-    enc->end = origend;
-    enc->offset = origoffset;
-
     Py_DECREF(labels);
     return ret;
 }
@@ -2429,7 +2420,6 @@ PyObject *objToJSON(PyObject *self, PyObject *args, PyObject *kwargs) {
     PRINTMARK();
     ret = JSON_EncodeObject(oinput, encoder, buffer, sizeof(buffer));
     PRINTMARK();
-
     if (PyErr_Occurred()) {
         PRINTMARK();
         return NULL;


Code review comments:
diff_hunk: @@ -1614,68 +1629,44 @@ char **NpyArr_encodeLabels(PyArrayObject *labels, JSONObjectEncoder *enc,
         ret[i] = NULL;
     }
 
-    origst = enc->start;
-    origend = enc->end;
-    origoffset = enc->offset;
-
     stride = PyArray_STRIDE(labels, 0);
     dataptr = PyArray_DATA(labels);
     getitem = (PyArray_GetItemFunc *)PyArray_DESCR(labels)->f->getitem;
     type_num = PyArray_TYPE(labels);
 
     for (i = 0; i < num; i++) {
-        if (PyTypeNum_ISDATETIME(type_num) || PyTypeNum_ISNUMBER(type_num))
-        {
-            item = (PyObject *)labels;
-            pyenc->npyType = type_num;
-            pyenc->npyValue = dataptr;
-        } else {
-            item = getitem(dataptr, labels);
-            if (!item) {
-                NpyArr_freeLabels(ret, num);
-                ret = 0;
-                break;
-            }
-        }
-
-        cLabel = JSON_EncodeObject(item, enc, labelBuffer, NPY_JSON_BUFSIZE);
-
-        if (item != (PyObject *)labels) {
-            Py_DECREF(item);
-        }
-
-        if (PyErr_Occurred() || enc->errorMsg) {
+        item = getitem(dataptr, labels);
+        if (!item) {
+	  NpyArr_freeLabels(ret, num);
+	  ret = 0;
+	  break;
+	}
+
+	PyObject *str = PyObject_Str(item);
path: pandas/_libs/src/ujson/python/objToJSON.c
text: @author:
@user1 last comment...here is where I think that check needs to go. We are getting the `str` of the object but when passed in as a DTA the introspection is different than when passed in an object array. Wondering if we have shared utilities to already handle that ambiguity I could leverage instead of trying to do it all in this extension