pydata/pandas — pull request #9130 by Komnomnomnom (2014-12-22 16:13:48+00:00)
Title: PERF: json support for blocks GH9037

This adds block support to the JSON serialiser, as per #9037. I also added code to directly cast and serialise numpy data, which replaces the previous use of intermediate Python objects.

Large performance improvement (~25x) for mixed frames containing datetimes / timedeltas.

```
-------------------------------------------------------------------------------
Test name | head[ms] | base[ms] | ratio |
-------------------------------------------------------------------------------
packers_write_json_mixed_delta_int_tstamp | 97.1374 | 2633.7843 | 0.0369 |
packers_write_json_mixed_float_int_T | 68.7390 | 86.4150 | 0.7955 |
packers_write_json_date_index | 68.9886 | 83.2930 | 0.8283 |
packers_write_json_T | 61.1283 | 72.9477 | 0.8380 |
packers_write_json | 60.7293 | 71.3053 | 0.8517 |
packers_read_json_date_index | 157.4903 | 161.6703 | 0.9741 |
packers_read_json | 157.4220 | 157.8477 | 0.9973 |
packers_write_json_mixed_float_int | 98.1897 | 98.1696 | 1.0002 |
packers_write_json_mixed_float_int_str | 84.0390 | 83.6937 | 1.0041 |
-------------------------------------------------------------------------------
Test name | head[ms] | base[ms] | ratio |
-------------------------------------------------------------------------------
```

Some questions, any comments appreciated:
1. there's a little overhead to dealing with blocks so I'm avoiding using them and using `values` instead if the frame is 'simple'. I'm using the BlockManager methods `_is_single_block` and `is_mixed_dtype` for this to check for a simple frame.
2. I'm using the DataFrame `_data` attr and `mgr_locs` to get access to the block data and the block-to-column mapping. Are there any caveats to this? I know the DataFrame does some caching, but I'm not familiar enough with the details.

Tested locally on Python 2.7 for 32 & 64 bit linux and 3.3. on 64 bit linux. JSON tests run through valgrind. Would appreciate if someone could give it a bash on Windows before merging.

@cpcloud I also fixed a ref leak and added support for `date_unit` in the #9028 code.