Repository: pandas-dev/pandas

Issue:
df.to_json() slower in 0.13.x vs 0.12.0
df.to_json() method seems consistently ~1.8x slower in version 0.13.x (and a few older 0.12.x versions in the master branch on git) than in 0.12.0.

Version 0.12.0:

``` python
Python 2.7.5+ (default, Sep 17 2013, 15:31:50) 
In [1]: import pandas as pd, numpy as np

In [2]: df = pd.DataFrame(np.random.rand(100000,10))

In [3]: %timeit df.to_json(orient='split')
10 loops, best of 3: 96.1 ms per loop

In [4]: pd.__version__, np.__version__
Out[4]: ('0.12.0', '1.7.1') 
```

Version 0.13.0rc1:

``` python
Python 2.7.5+ (default, Sep 17 2013, 15:31:50) 
In [1]: import pandas as pd, numpy as np

In [2]: df = pd.DataFrame(np.random.rand(100000,10))

In [3]: %timeit df.to_json(orient='split')
10 loops, best of 3: 172 ms per loop

In [4]: pd.__version__, np.__version__
Out[4]: ('0.13.0rc1-119-g2485e09', '1.8.0')
```

The 1.8x factor seems to hold on my machine across Python versions (2.7.5 vs 3.3.2), dataframe sizes, orient values and dtypes (only tried floats and DatetimeIndex).

Was there some change in to_json() or have I goofed something up in my environment?



Discussion:
cc @Komnomnomnom

Should I be worried that it takes me ~700ms to do the same thing, or is the original computer super-fast?

as an aside....could prob add a vbench (or 2) in `pandas/vb/packers.py` (to at least track this from version to version)

I see similar results, ~200ms 0.12 vs. ~430ms 0.13rc

I ran a little `to_json()` timing test over a bunch of commits, and I believe 2 slowdowns are being introduced since 0.12, a smaller one and a larger one.  I'm fairly sure the larger slowdown is this commit: https://github.com/pydata/pandas/commit/edefe981ca2422e78045b981bb373cfbeb458097

In my testing `to_json()` takes 1.5x as long in this commit vs its parent.  I honestly don't know enough about the tradeoffs of locale vs slowdown to submit any sort of pull request to fix it, but at least that should help a future person narrow down this bug.

Thanks guys, I'm going to take a look into this one and see if I can improve / remove the slowdown, haven't had much time over the Christmas period. Thanks @danbirken for narrowing it down :), got a code snippet for the tests you're running?

Very similar to the sample in the OP (I just added a hash to ensure the results were identical... they are):

```
import pandas as pd
import hashlib, time

df = pd.DataFrame({'a': range(500000), 'b': [i / 1000. for i in range(500000)]})

total_time = 0

for i in range(10):
    start = time.time()
    out = df.to_json()
    total_time += time.time() - start

print 'Time: %.2fs, Hash: %s' % (
    total_time, hashlib.md5(out).hexdigest()
)
```

Then:

```
$ git checkout c7b578c2176c9a0e1099b07d9b01be25c6fa3bca
$ python setup.py clean
$ python setup.py build_ext --inplace
$ python timing_test.py
Time: 4.49s, Hash: 2608e38a63003e637a3a108760bdf46e

$ git checkout edefe981ca2422e78045b981bb373cfbeb458097
$ python setup.py clean
$ python setup.py build_ext --inplace
$ python timing_test.py
Time: 6.42s, Hash: 2608e38a63003e637a3a108760bdf46e
```

@Komnomnomnom anything on this?

hey @jreback, should have some time this weekend

np

FYI I looked into this a bit more, and while it does seem the (my) locale code is slowing things down a bit, the main culprit is much earlier. For code jsonifying a dataframe of 100,000 floats vbench produces

![download](https://f.cloud.github.com/assets/107748/2003862/bbd663f2-867e-11e3-9481-08f9a76e63d3.png)

which points to this PR (mine!) #4498. I'll see if I can sort it out.


Resolving patch:
diff --git a/vb_suite/packers.py b/vb_suite/packers.py
--- a/vb_suite/packers.py
+++ b/vb_suite/packers.py
@@ -92,3 +92,23 @@ def remove(f):
 
 packers_write_hdf_table = Benchmark("df.to_hdf(f,'df',table=True)", setup, cleanup="remove(f)", start_date=start_date)
 
+#----------------------------------------------------------------------
+# json
+
+setup_int_index = """
+import numpy as np
+df.index = np.arange(50000)
+"""
+
+setup = common_setup + """
+df.to_json(f,orient='split')
+"""
+packers_read_json_date_index = Benchmark("pd.read_json(f, orient='split')", setup, start_date=start_date)
+setup = setup + setup_int_index
+packers_read_json = Benchmark("pd.read_json(f, orient='split')", setup, start_date=start_date)
+
+setup = common_setup + """
+"""
+packers_write_json_date_index = Benchmark("df.to_json(f,orient='split')", setup, cleanup="remove(f)", start_date=start_date)
+setup = setup + setup_int_index
+packers_write_json = Benchmark("df.to_json(f,orient='split')", setup, cleanup="remove(f)", start_date=start_date)


Test patch: