Repository: pandas-dev/pandas

Issue:
Json import sorting
Noticed that if export a df to json file and then reload the data, ordering is not kept.

``` python
import pandas as pd

df.to_json('test.json')
newdf = pd.read_json('test.json')

print df.index
```

gives:

``` python
1
10
100
....
```

instead of 

``` python
1
2
3
....
```

it suffices to call `newdf = pd.read_json('test.json').sort()` to have the what I'd expect.

Is this intended behaviour or bug?



Discussion:
The JSON emitted is a dictionary object, not an array, so it's inherently
unordered. It's not possible to maintain ordering (and it's outside of the
JSON spec to do so).

no it's sorting lexicographically, most likely because the index is sorted before being converted to a numeric dtype. I think this is a bug.

@cpcloud sure, that may be, but fundamentally, you can't assure that whatever ordering was there initially should be in the final index.  I.e.:

``` python
df = DataFrame([range(10)] * 10, index = range(10,0,-1))
df.from_json(df.to_json())
```

There's no way that you can ensure that the index falls out correctly. But yes, it's clearly lexicographically sorting.

hm....okay...but why is it sorting at all?

The JSON code initially decodes to a dict with string keys which it then passes to the `DataFrame` constructor, I think the sorting might be happening after that point:

``` python
In [9]: d = {'foo': {'1': 1, '2': 2, '3': 3, '10': 10, '15': 15, '100': 100, '101': 101}}

In [10]: pd.DataFrame(d)
Out[10]: 
     foo
1      1
10    10
100  100
101  101
15    15
2      2
3      3

```

Yep, that'll do it! `dict`s are sorted in the `DataFrame` constructor.

passing a list-of-lists will not sort (though may be more expensive 2 create)

BTW @ghego if you want to ensure that order is preserved during roudtrip JSONifying try using `orient=split`

``` python
n [27]: pd.read_json(df.to_json(orient='split'), orient='split')
[0]
Out[27]: 
     foo
1      1
2      2
3      3
4     10
10    11
11    20
100  100
101  101

```

Might be nice to have that in the docs/cookbook.

@cpcloud agreed. There's a couple of things I'd like to add to the JSON docs (incl a couple of benchmarks and more about the numpy param and when it can be useful). I'll try and get a PR together for this stuff before the next release. 

So I think this can be closed because it's expected behavior?

I think @Komnomnomnom going to take a look....so let's leave open for a bit

Just to clarify I'm going to update the docs to give more detail on the different orients, and make it clear when you can expect non order-preserving behaviour. Apart from that I wasn't going to change anything.... 

@Komnomnomnom changing this to docs only....when u have a chance

@Komnomnomnom how's docs coming on this?

@Komnomnomnom do we need anything in docs for this? (I believe you did cover it...)...lmk

@jreback, sorry yeah still planning to get to the json docs at some point. Hopefully will have some time over the next few days.

gr8

@Komnomnomnom docs?

I've done a rough pass but still struggling to find adequate time I'm afraid. Last chance this weekend? :)

docs are ok even after release-candidate....so ok


Resolving patch:
diff --git a/pandas/io/json.py b/pandas/io/json.py
--- a/pandas/io/json.py
+++ b/pandas/io/json.py
@@ -153,8 +153,9 @@ def read_json(path_or_buf=None, orient=None, typ='frame', dtype=True,
     keep_default_dates : boolean, default True.
         If parsing dates, then parse the default datelike columns
     numpy : boolean, default False
-        Direct decoding to numpy arrays. Note that the JSON ordering MUST be
-        the same for each term if numpy=True.
+        Direct decoding to numpy arrays. Supports numeric data only, but
+        non-numeric column and index labels are supported. Note also that the
+        JSON ordering MUST be the same for each term if numpy=True.
     precise_float : boolean, default False.
         Set to enable usage of higher precision (strtod) function when
         decoding string to double values. Default (False) is to use fast but


Test patch: