BUG: from_dict ignored order of OrderedDict (#8425)
State: closed

- [x] closes #8425
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Updated some existing test codes because index order in some DataFrames constructed by from_dict have changed after this fix.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: DataFrame
description: DataFrame data structure

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26875?src=pr&el=h1) Report
> Merging [#26875](https://codecov.io/gh/pandas-dev/pandas/pull/26875?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/430f0fd04646a0ab44ccbfbad706dfe22e9fabfe?src=pr&el=desc) will **decrease** coverage by `1.41%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26875/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26875?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26875      +/-   ##
==========================================
- Coverage   91.88%   90.46%   -1.42%     
==========================================
  Files         179      179              
  Lines       50696    50699       +3     
==========================================
- Hits        46581    45867     -714     
- Misses       4115     4832     +717
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.46% <100%> (ø)` | :arrow_up: |
| #single | `?` | |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26875?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/api.py](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9hcGkucHk=) | `99% <100%> (ø)` | :arrow_up: |
| [pandas/core/internals/construction.py](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2NvbnN0cnVjdGlvbi5weQ==) | `95.96% <100%> (+0.03%)` | :arrow_up: |
| [pandas/core/computation/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vcHl0YWJsZXMucHk=) | `62.5% <0%> (-27.75%)` | :arrow_down: |
| [pandas/io/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3B5dGFibGVzLnB5) | `63.82% <0%> (-26.48%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |
| [pandas/core/computation/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vY29tbW9uLnB5) | `84.21% <0%> (-5.27%)` | :arrow_down: |
| [pandas/core/computation/expr.py](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vZXhwci5weQ==) | `94.78% <0%> (-3.03%)` | :arrow_down: |
| [pandas/io/clipboard/clipboards.py](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NsaXBib2FyZC9jbGlwYm9hcmRzLnB5) | `31.88% <0%> (-2.9%)` | :arrow_down: |
| [pandas/io/formats/printing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvcHJpbnRpbmcucHk=) | `84.49% <0%> (-1.07%)` | :arrow_down: |
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `96.21% <0%> (-0.17%)` | :arrow_down: |
| ... and [3 more](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26875?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26875?src=pr&el=footer). Last update [430f0fd...d765439](https://codecov.io/gh/pandas-dev/pandas/pull/26875?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26875?src=pr&el=h1) Report
> Merging [#26875](https://codecov.io/gh/pandas-dev/pandas/pull/26875?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/2efb60717bda9fc64344c5f6647d58564930808e?src=pr&el=desc) will **decrease** coverage by `0.8%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26875/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26875?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26875      +/-   ##
==========================================
- Coverage   92.79%   91.99%   -0.81%     
==========================================
  Files         180      180              
  Lines       50417    50776     +359     
==========================================
- Hits        46784    46709      -75     
- Misses       3633     4067     +434
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.63% <100%> (-0.84%)` | :arrow_down: |
| #single | `41.83% <100%> (-0.6%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26875?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/internals/construction.py](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2NvbnN0cnVjdGlvbi5weQ==) | `95.97% <100%> (-0.78%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |
| [pandas/io/excel/\_openpyxl.py](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2V4Y2VsL19vcGVucHl4bC5weQ==) | `84.71% <0%> (-3.23%)` | :arrow_down: |
| [pandas/core/internals/blocks.py](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2Jsb2Nrcy5weQ==) | `94.38% <0%> (-1.01%)` | :arrow_down: |
| [pandas/core/tools/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdG9vbHMvZGF0ZXRpbWVzLnB5) | `85.05% <0%> (-0.74%)` | :arrow_down: |
| [pandas/core/arrays/integer.py](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2ludGVnZXIucHk=) | `96.3% <0%> (-0.59%)` | :arrow_down: |
| [pandas/io/formats/printing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvcHJpbnRpbmcucHk=) | `86.72% <0%> (-0.48%)` | :arrow_down: |
| [pandas/core/dtypes/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2NvbmNhdC5weQ==) | `96.58% <0%> (-0.46%)` | :arrow_down: |
| [pandas/core/config\_init.py](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29uZmlnX2luaXQucHk=) | `95.8% <0%> (-0.4%)` | :arrow_down: |
| ... and [154 more](https://codecov.io/gh/pandas-dev/pandas/pull/26875/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26875?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26875?src=pr&el=footer). Last update [2efb607...b379d8a](https://codecov.io/gh/pandas-dev/pandas/pull/26875?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

not sure what caused this error. 
I’ll work on this later tonight.
Azure pipelines raised some test errors in Python 3.5 and I checked some of them. The change in this PR made the difference in order of DataFrame index if it is constructed from a dict. I suppose this is expected behavior, because `from_dict` no longer sorts index and a dict is not ordered in Python 3.5.
@TomAugspurger, thank you.
I followed your instruction to create a new env, but it raised below error when importing pandas.
```Python
>>> import pandas as pd
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/masayo/Documents/work/contributions/pandas/pandas/__init__.py", line 25, in <module>
    from pandas._libs import (hashtable as _hashtable,
  File "/Users/masayo/Documents/work/contributions/pandas/pandas/_libs/__init__.py", line 3, in <module>
    from .tslibs import (
  File "/Users/masayo/Documents/work/contributions/pandas/pandas/_libs/tslibs/__init__.py", line 3, in <module>
    from .conversion import normalize_date, localize_pydatetime
  File "__init__.pxd", line 918, in init pandas._libs.tslibs.conversion
ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject
```


FYR, I created another env and installed Python 3.5.6 and pandas (`python -m pip install -e`). `expected` in test function `test_concat_tuple_keys` returns a DataFrame whose index is not sorted.
```Python
>>> expected = pd.DataFrame(
...             {'A': {('bee', 'bah', 0): 1.0,
...                    ('bee', 'bah', 1): 1.0,
...                    ('bee', 'boo', 0): 2.0,
...                    ('bee', 'boo', 1): 2.0,
...                    ('bee', 'boo', 2): 2.0},
...              'B': {('bee', 'bah', 0): 1.0,
...                    ('bee', 'bah', 1): 1.0,
...                    ('bee', 'boo', 0): 2.0,
...                    ('bee', 'boo', 1): 2.0,
...                    ('bee', 'boo', 2): 2.0}})
>>> expected
             A    B
bee boo 1  2.0  2.0
        2  2.0  2.0
    bah 1  1.0  1.0
        0  1.0  1.0
    boo 0  2.0  2.0
```
> I followed your instruction to create a new env, but it raised below error when importing pandas.

Not sure, but if you're in the same git repo, you would need to rebuild the C extensions for that new environment. Locally I have my main `pandas` git repo, and a second `pandas-35` repo.

> returns a DataFrame whose index is not sorted.

Can you clarify: is that on this branch? When I run that in a 3.5 env on master (from a few weeks ago) I get

```python
Python 3.5.6 |Anaconda, Inc.| (default, Aug 26 2018, 16:30:03)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.DataFrame(
...  {'A': {('bee', 'bah', 0): 1.0,
...         ('bee', 'bah', 1): 1.0,
...         ('bee', 'boo', 0): 2.0,
...         ('bee', 'boo', 1): 2.0,
...         ('bee', 'boo', 2): 2.0},
...   'B': {('bee', 'bah', 0): 1.0,
...         ('bee', 'bah', 1): 1.0,
...         ('bee', 'boo', 0): 2.0,
...         ('bee', 'boo', 1): 2.0,
...         ('bee', 'boo', 2): 2.0}})
             A    B
bee bah 0  1.0  1.0
        1  1.0  1.0
    boo 0  2.0  2.0
        1  2.0  2.0
        2  2.0  2.0
```
> > returns a DataFrame whose index is not sorted.
> 
> Can you clarify: is that on this branch? 

Yes it is. No change have been made after commit ae19e6c.  

You’re right. I didn’t rebuild pandas after merging master.
But even after `python setup.py build_ext --inplace -j 4`, I still get below error when importing pandas. Maybe it it because the files (pandas
/_libs/tslibs/c_timestamp.*) related to this error is not updated after the build. Could you tell me how to full build?
```Python
>>> import pandas as pd
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/masayo/Documents/work/contributions/pandas/pandas/__init__.py", line 25, in <module>
    from pandas._libs import (hashtable as _hashtable,
  File "/Users/masayo/Documents/work/contributions/pandas/pandas/_libs/__init__.py", line 3, in <module>
    from .tslibs import (
  File "/Users/masayo/Documents/work/contributions/pandas/pandas/_libs/tslibs/__init__.py", line 3, in <module>
    from .conversion import normalize_date, localize_pydatetime
  File "pandas/_libs/tslibs/c_timestamp.pxd", line 7, in init pandas._libs.tslibs.conversion
    cdef class _Timestamp(datetime):
  File "__init__.pxd", line 918, in init pandas._libs.tslibs.c_timestamp
ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject
```
I think `python setup.py clean` will do the trick.

On Tue, Jun 25, 2019 at 4:33 PM mazayo <notifications@github.com> wrote:

> returns a DataFrame whose index is not sorted.
>
> Can you clarify: is that on this branch?
>
> Yes it is. No change have been made after commit ae19e6c
> <https://github.com/pandas-dev/pandas/commit/ae19e6c850f13daacdf018a1b06b282ef72d3f67>
> .
>
> You’re right. I didn’t rebuild pandas after merging master.
> But even after python setup.py build_ext --inplace -j 4, I still get
> below error when importing pandas. Maybe it it because the files (pandas
> /_libs/tslibs/c_timestamp.*) related to this error is not updated after
> the build. Could you tell me how to full build?
>
> >>> import pandas as pd
>
> Traceback (most recent call last):
>
>   File "<stdin>", line 1, in <module>
>
>   File "/Users/masayo/Documents/work/contributions/pandas/pandas/__init__.py", line 25, in <module>
>
>     from pandas._libs import (hashtable as _hashtable,
>
>   File "/Users/masayo/Documents/work/contributions/pandas/pandas/_libs/__init__.py", line 3, in <module>
>
>     from .tslibs import (
>
>   File "/Users/masayo/Documents/work/contributions/pandas/pandas/_libs/tslibs/__init__.py", line 3, in <module>
>
>     from .conversion import normalize_date, localize_pydatetime
>
>   File "pandas/_libs/tslibs/c_timestamp.pxd", line 7, in init pandas._libs.tslibs.conversion
>
>     cdef class _Timestamp(datetime):
>
>   File "__init__.pxd", line 918, in init pandas._libs.tslibs.c_timestamp
> ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/26875?email_source=notifications&email_token=AAKAOIUG7S3E6AHWVYUOYTTP4KFKBA5CNFSM4HYQIEH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYRVDJI#issuecomment-505631141>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIUYAMYSLZSB5O74Q6DP4KFKBANCNFSM4HYQIEHQ>
> .
>

Thank you. I could successfully full build pandas by `python setup.py clean` followed by `python setup.py build_ext --inplace -j 4`. 

Here is what I got in this branch and in the the environment created by `conda env create -n pandas-azure-macos-35  --file=ci/deps/azure-macos-35.yaml`.
```Python
>>> import pandas as pd
>>> pd.__version__
'0.25.0.dev0+784.gae19e6c85'
>>> expected = pd.DataFrame(
...             {'A': {('bee', 'bah', 0): 1.0,
...                    ('bee', 'bah', 1): 1.0,
...                    ('bee', 'boo', 0): 2.0,
...                    ('bee', 'boo', 1): 2.0,
...                    ('bee', 'boo', 2): 2.0},
...              'B': {('bee', 'bah', 0): 1.0,
...                    ('bee', 'bah', 1): 1.0,
...                    ('bee', 'boo', 0): 2.0,
...                    ('bee', 'boo', 1): 2.0,
...                    ('bee', 'boo', 2): 2.0}})
>>> expected
             A    B
bee boo 1  2.0  2.0
        2  2.0  2.0
    bah 1  1.0  1.0
        0  1.0  1.0
    boo 0  2.0  2.0
>>>
>>>
>>> import numpy as np
>>> df1 = pd.DataFrame(np.ones((2, 2)), columns=list('AB'))
>>> df2 = pd.DataFrame(np.ones((3, 2)) * 2, columns=list('AB'))
>>> results = pd.concat((df1, df2), keys=[('bee', 'bah'), ('bee', 'boo')])
>>> results
             A    B
bee bah 0  1.0  1.0
        1  1.0  1.0
    boo 0  2.0  2.0
        1  2.0  2.0
        2  2.0  2.0
```

Sorry it took me a while to get back to you. I was about to go work when I received you message.
OK, does that output make sense to out? I haven't looked to closely, but it seems like the old dict-of-dicts constructor sorted unordered dictionaries?
Right. `extract_index` is called when constructing a DataFrame from a dict. It used to sort keys before the update in this branch.
thanks @mazayo 
Thanks for offering good first issues here and your support