Repository: pandas-dev/pandas

Issue:
BUG: DataFrame from_dict constructor ignores Ordered dict when orient='index'
Hello, 
I have been experimenting with OrderedDicts lately, and found a bug with the DataFrame from_dict constructor. Here is a sample code.

``` python
import collections
import pandas as pd

firstrow={}
firstrow['foo'] = 'bar'
firstrow['baz'] = 'buzz'

row1 = pd.Series(firstrow)

secondrow={}
secondrow['foo'] = 'bar2'
secondrow['baz'] = 'buzz2'

row2 = pd.Series(secondrow)

roworder = collections.OrderedDict()

roworder['zShould be first'] = row1
roworder['Should be second'] = row2

# Ordering is respected when sorting on columns
df = pd.DataFrame.from_dict(data=roworder, orient='columns')

# But not when sorting on rows
incorrectdf = pd.DataFrame.from_dict(data=roworder, orient='index')
correctdf = df.transpose()
```
## INSTALLED VERSIONS

commit: None
python: 3.3.5.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 26 Stepping 5, GenuineIntel
byteorder: little
LC_ALL: None
LANG: fr_CH

pandas: 0.14.1
nose: 1.3.4
Cython: 0.20.1
numpy: 1.9.0
scipy: 0.13.3
statsmodels: 0.5.0
IPython: 2.2.0
sphinx: 1.2.3
patsy: 0.3.0
scikits.timeseries: None
dateutil: 2.2
pytz: 2013.9
bottleneck: None
tables: 3.1.1
numexpr: 2.4
matplotlib: 1.4.0
openpyxl: None
xlrd: 0.9.3
xlwt: None
xlsxwriter: 0.5.7
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
rpy2: None
sqlalchemy: 0.9.4
pymysql: None
psycopg2: None



Discussion:
can you make your code runnable (so can simply copy/paste). you have some undefined variables.

Sorry about that! Should be fine now. If not, will check when back in the office tomorrow. 

EDIT: the code now reproduces the above mentioned bug

@Gimli510 that does look buggy.

welcome a pull-request to fix.

You can use your test example above, just step thru the code and see where its breaking and try a fix.

@jreback I think I found where the bug comes from.
The function _union_index calls
lib.fast_unique_multiple_list(indexes), which sorts the keys before returning them. Should we carry a flag telling this cython function not to sort the keys when the indexes list was created from an ordered dict? I guess there is a cleaner way to do this, but don't really have any idea about how to go about it.

``` python
# Up to this point, the future index is ordered as it should.
indexes = [['zShould be first', 'Should be second'], ['zShould be first', 'Should be second']]
# When indexes is a list with more than 1 items, we hit this path:        
# return Index(lib.fast_unique_multiple_list(indexes))

# However, 
lib.fast_unique_multiple_list(indexes)
```

returns

``` python
['Should be second', 'zShould be first']
```

I think this should be handled in `core/pandas/frame/extract_index`. Need to differentiate between a dict and an OrderedDict. 

maybe add in a `have_ordered` in addition to setting `have_dict`. Then you can pass this to `_union_indexes(indexes,ordered=have_ordered)`

Then you can validate that if `ordered=True` is passed (default is False)
then can do a unique preserving order (so pass the flag into `fast_unique_multiple`, iow don't sort)

@jreback 
I have done based on what you said  and in the last part how can I pass the flag to fast_unique_multiple because it calls fast_unique_multiple_list(_args, *_kwargs) and when I look at the lib.pyx it always sort the list at the end(uniques.sort())
any idea?

@jreback is this still an issue in the current version of `pandas`? I'm seeing the problem on an older version (v0.16.2) and I'm not sure if it's been addressed in the current one.

```python
df = pd.DataFrame.from_dict(ordered_dict_data, orient='index') 
```
sorts the index alphabetically. I've been using the following hack to address it:

```python
df = pd.DataFrame.from_dict(ordered_dict_data, orient='columns').T
```
My hack, however, sorts the columns alphabetically.

For the data that I have, it's easier for me to re-order these columns so the latter solution works better. To be precise, my `data` is an `OrderedDict` of `OrderedDict`s so I expect the sort order of both the index and columns to be respected. It looks something like this:

```python
data = OrderedDict(
    'a': OrderedDict('aa': 5, 'bb': 10),
    'b': OrderedDict('aa': 7, 'bb': 14),
    ...)
```

If it's not fixed, I can take a stab at it.
Still an open issue.

Resolving patch:
diff --git a/pandas/core/internals/construction.py b/pandas/core/internals/construction.py
--- a/pandas/core/internals/construction.py
+++ b/pandas/core/internals/construction.py
@@ -9,6 +9,7 @@
 
 from pandas._libs import lib
 from pandas._libs.tslibs import IncompatibleFrequency, OutOfBoundsDatetime
+import pandas.compat as compat
 from pandas.compat import raise_with_traceback
 
 from pandas.core.dtypes.cast import (
@@ -338,6 +339,7 @@ def extract_index(data):
         have_raw_arrays = False
         have_series = False
         have_dicts = False
+        have_ordered = False
 
         for val in data:
             if isinstance(val, ABCSeries):
@@ -345,6 +347,8 @@ def extract_index(data):
                 indexes.append(val.index)
             elif isinstance(val, dict):
                 have_dicts = True
+                if isinstance(val, OrderedDict):
+                    have_ordered = True
                 indexes.append(list(val.keys()))
             elif is_list_like(val) and getattr(val, "ndim", 1) == 1:
                 have_raw_arrays = True
@@ -353,8 +357,10 @@ def extract_index(data):
         if not indexes and not raw_lengths:
             raise ValueError("If using all scalar values, you must pass" " an index")
 
-        if have_series or have_dicts:
+        if have_series:
             index = _union_indexes(indexes)
+        elif have_dicts:
+            index = _union_indexes(indexes, sort=not (compat.PY36 or have_ordered))
 
         if have_raw_arrays:
             lengths = list(set(raw_lengths))


Test patch:
diff --git a/pandas/tests/frame/test_constructors.py b/pandas/tests/frame/test_constructors.py
--- a/pandas/tests/frame/test_constructors.py
+++ b/pandas/tests/frame/test_constructors.py
@@ -517,7 +517,8 @@ def test_constructor_subclass_dict(self, float_frame):
             dct.update(v.to_dict())
             data[k] = dct
         frame = DataFrame(data)
-        tm.assert_frame_equal(float_frame.sort_index(), frame)
+        expected = frame.reindex(index=float_frame.index)
+        tm.assert_frame_equal(float_frame, expected)
 
     def test_constructor_dict_block(self):
         expected = np.array([[4.0, 3.0, 2.0, 1.0]])
@@ -1203,7 +1204,7 @@ def test_constructor_list_of_series(self):
 
         sdict = OrderedDict(zip(["x", "Unnamed 0"], data))
         expected = DataFrame.from_dict(sdict, orient="index")
-        tm.assert_frame_equal(result.sort_index(), expected)
+        tm.assert_frame_equal(result, expected)
 
         # none named
         data = [
@@ -1342,7 +1343,7 @@ def test_constructor_list_of_namedtuples(self):
     def test_constructor_orient(self, float_string_frame):
         data_dict = float_string_frame.T._series
         recons = DataFrame.from_dict(data_dict, orient="index")
-        expected = float_string_frame.sort_index()
+        expected = float_string_frame.reindex(index=recons.index)
         tm.assert_frame_equal(recons, expected)
 
         # dict of sequence
@@ -1351,6 +1352,19 @@ def test_constructor_orient(self, float_string_frame):
         xp = DataFrame.from_dict(a).T.reindex(list(a.keys()))
         tm.assert_frame_equal(rs, xp)
 
+    def test_constructor_from_ordered_dict(self):
+        # GH8425
+        a = OrderedDict(
+            [
+                ("one", OrderedDict([("col_a", "foo1"), ("col_b", "bar1")])),
+                ("two", OrderedDict([("col_a", "foo2"), ("col_b", "bar2")])),
+                ("three", OrderedDict([("col_a", "foo3"), ("col_b", "bar3")])),
+            ]
+        )
+        expected = DataFrame.from_dict(a, orient="columns").T
+        result = DataFrame.from_dict(a, orient="index")
+        tm.assert_frame_equal(result, expected)
+
     def test_from_dict_columns_parameter(self):
         # GH 18529
         # Test new columns parameter for from_dict that was added to make