PERF: json support for blocks GH9037
State: closed

This adds block support to the JSON serialiser, as per #9037. I also added code to directly cast and serialise numpy data, which replaces the previous use of intermediate Python objects.

Large performance improvement (~25x) for mixed frames containing datetimes / timedeltas. 

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
packers_write_json_mixed_delta_int_tstamp    |  97.1374 | 2633.7843 |   0.0369 |
packers_write_json_mixed_float_int_T         |  68.7390 |  86.4150 |   0.7955 |
packers_write_json_date_index                |  68.9886 |  83.2930 |   0.8283 |
packers_write_json_T                         |  61.1283 |  72.9477 |   0.8380 |
packers_write_json                           |  60.7293 |  71.3053 |   0.8517 |
packers_read_json_date_index                 | 157.4903 | 161.6703 |   0.9741 |
packers_read_json                            | 157.4220 | 157.8477 |   0.9973 |
packers_write_json_mixed_float_int           |  98.1897 |  98.1696 |   1.0002 |
packers_write_json_mixed_float_int_str       |  84.0390 |  83.6937 |   1.0041 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
```

Some questions, any comments appreciated:
1. there's a little overhead to dealing with blocks so I'm avoiding using them and using `values` instead if the frame is 'simple'. I'm using the BlockManager methods `_is_single_block` and `is_mixed_dtype` for this to check for a simple frame.
2. I'm using the DataFrame `_data` attr and `mgr_locs` to get access to the block data and the block-to-column mapping. Are there any caveats to this? I know the DataFrame does some caching, but I'm not familiar enough with the details.

Tested locally on Python 2.7 for 32 & 64 bit linux and 3.3. on 64 bit linux. JSON tests run through valgrind. Would appreciate if someone could give it a bash on Windows before merging.

@cpcloud I also fixed a ref leak and added support for `date_unit` in the #9028 code.


Labels:
name: Performance
description: Memory or execution speed performance
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
@Komnomnomnom 

1) you can really just use `_is_mixed_type`. `_is_single_block` distinguished between a ndim==1 and ndim>1 (e.g. a Series and DataFrame). 

The key is that `.values` on a non-mixed_type (e.g. a Series or a single-dtyped DataFrames) is free (no copyin data), and you will get back a single dtype.

2) `_data` is guaranteed to exists on a `PandasObject` (e.g. Series/DataFrame) as well as `mgr_locs` (which is also correct), this handles the position in the block to the column names mappings.

lots of code changes! I will give a test on windows and let you know.

Looks good though.

Has the serialization order changed at all? Does it matter if it does? IOW for some formats the orderings might be different now.

Thanks @jreback, I used `._data.mgr_locs` rather than `.blocks` so I could preserve the column order (also `.blocks` on the frame returns a copy I think?). 

I have not modified (or added) tests, which should enforce that serialisation order has not changed. Maybe I'll add a compat test though, just to be sure.

FWIW the valgrind run for these changes is clean, (but I'll be submitting a PR soon to fix an unrelated segfault in the code for handling Python datetime.time objects)

this passes everythng for me on windows, so good to go for me. lmk when you are satisfied with tests.

aside: https://github.com/pydata/pandas/blob/master/pandas/io/tests/test_json/test_ujson.py#L120 fails on windows (on master), but I wonder if its just a precision in the test issue (its like 1 part in 100 digits close), so it passes an `.allclose` test

cc @cpcloud 

+1000 here nice work. Sorry about the leak! 

@Komnomnomnom

What are your thoughts on a round-trippable json orient (maybe "roundtrip")? Ie provide enough metadata to reconstruct a frame or series with 100% fidelity.

Alternatively how about exposing dumps at the toplevel, and folks could roll their own.

@cpcloud a roundtrip orient sounds like a good idea, there was some related discussion in #4889 but I'm not keen on adding metadata into all the orients.  But I like the notion of a new orient with the same format as `split` but an additional `_meta` entry with info on blocks and their dtypes. Would it need any other info to roundtrip properly? I've been thinking about a couple more orients too (from #8333 and #5729).

I also like the idea of exposing json on the top level, as you can easily give it more than just frames and series types, i.e. it will happily process numpy arrays, pandas indices and other Python types quite efficiently. Maybe `pd.to_json` ?

@jreback that windows issue is weird, what do you get when you try:

``` python
In [6]: import pandas.json as ujson

In [7]: ujson.encode(1e-100)
Out[7]: '1e-100'

In [8]: ujson.decode('1e-100')
Out[8]: 1e-100

```

Ok I've added a compat test for completeness, removed the call to `_is_single_block` and updated the release notes. 

The only compat issue I can think of is that the json output for mixed frames will be slightly different due to integers being promoted to floats previously.

e.g. v0.15.2

``` python
In [2]: pd.DataFrame({'i': [1,2], 'f': [3.0, 4.2]}).to_json()
Out[2]: '{"f":{"0":3.0,"1":4.2},"i":{"0":1.0,"1":2.0}}'
```

this PR

``` python
In [3]:  pd.DataFrame({'i': [1,2], 'f': [3.0, 4.2]}).to_json()
Out[3]: '{"f":{"0":3.0,"1":4.2},"i":{"0":1,"1":2}}'
```

I've added an entry in the release notes about this too.

@Komnomnomnom minor doc comment. ping when ready and we can merge.

@Komnomnomnom 

I tested 0.15.2 on win64 and

```
In [8]: ujson.decode('1e-100')
Out[8]: 1e-100
```

is indeed produced

with the current PR

`9.999999999999999999998e-101` is produced
very odd

Ok just tried this myself on win64. I used conda and [this article](http://www.menpo.io/blog/2014/07/python_windows_development/) to compile with msvc. I had to make a couple of code fixes to get it to compile and it worked fine for `ujson.decode('1e-100')` for me..... 

The code for deserialisation wasn't changed by this PR and hasn't changed since 0.15.2 I think. Might be an issue with your compiler? What are you using, mingw?

if you conda install libpython then u can use mingw out of the box to compile

so prob a compiler issue then
ok maybe should skip this test on windows (or at least that part)?

I'm not sure how conda is setup but I've had issues before with an extension compiled with mingw when the python core and other extensions were compiled with msvc.

it's abi compat
and works for me (I think older versions might have that issue)

ok no biggie then 

ok works fine with msvc.....

thanks @Komnomnomnom 

@Komnomnomnom @cpcloud If you are interested in what is mentioned above (the 'roundtrip' orient, or the exposing of `dumps` or a `to_json` in the toplevel namespave), maybe open a new issue for that? 

Thanks @jorisvandenbossche #9146 #9147