Repository: pandas-dev/pandas

Issue:
Unconsistent (random) behaviour of pd.concat with different indexes
#### Code Sample, a copy-pastable example if possible

The following code

```python
import pandas as pd

dfs_sq = []

dfs_sq.append(pd.DataFrame(index=[0, 'sess'], columns=range(1,3)))
for i in range(5):
    dfs_sq.append(pd.DataFrame(index=[0, 1, 'sess'], columns=range(1,3)))

df_sq = pd.concat(dfs_sq, axis=1)

assert df_sq.index[1] == 'sess', df_sq.index
```
... saved as ``dependable.py``, succeeds if called as ``PYTHONHASHSEED=40 python3 dependable.py`` and fails if called as ``PYTHONHASHSEED=41 python3 dependable.py``:
```bash
Traceback (most recent call last):
  File "dependable.py", line 11, in <module>
    assert df_sq.index[1] == 'sess', df_sq.index
AssertionError: Index([0, 1, 'sess'], dtype='object')
```
#### Problem description

This is not very nice.

#### Expected Output

The same - which one is not particularly important.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.5.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.0-3-amd64
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: it_IT.UTF-8
LOCALE: it_IT.UTF-8

pandas: 0.21.0.dev+389.g276f3089a
pytest: 3.0.6
pip: 9.0.1
setuptools: None
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.19.0
pyarrow: None
xarray: None
IPython: 5.1.0.dev
sphinx: 1.5.6
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.3.0
numexpr: 2.6.1
feather: 0.3.1
matplotlib: 2.0.2
openpyxl: None
xlrd: 1.0.0
xlwt: 1.1.2
xlsxwriter: 0.9.6
lxml: None
bs4: 4.5.3
html5lib: 0.999999999
sqlalchemy: 1.0.15
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: 0.2.1

</details>



Discussion:
Presumably this is because your Index is un-sortable, so the hash union defines the resulting order. 
 Not sure there's anything that can be done?

```python
In [16]: idx = dfs_sq[0].index.union(dfs_sq[1].index)

In [17]: idx
Out[17]: Index([0, 'sess', 1], dtype='object')

In [18]: idx.sort_values()
TypeError: '>' not supported between instances of 'int' and 'str'
```
> Not sure there's anything that can be done?

OK, I'm replying without having looked at the code, but: if two indexes are unsortable, then I expect the resulting union to respect their original order (with e.g. priority given to the first index if orders don't coincide). Actually, I would have expected this to happen even when the two indexes, and the union too, are sortable...
I'm also talking without having looked at much code ... but I believe we're currently doing a hash-based unique on the entire set of values, that's what tosses out the order.  I suppose we could do something more iterative like you're suggesting that would preserve it.

Resolving patch:
diff --git a/pandas/core/common.py b/pandas/core/common.py
--- a/pandas/core/common.py
+++ b/pandas/core/common.py
@@ -629,3 +629,17 @@ def _random_state(state=None):
     else:
         raise ValueError("random_state must be an integer, a numpy "
                          "RandomState, or None")
+
+
+def _get_distinct_objs(objs):
+    """
+    Return a list with distinct elements of "objs" (different ids).
+    Preserves order.
+    """
+    ids = set()
+    res = []
+    for obj in objs:
+        if not id(obj) in ids:
+            ids.add(id(obj))
+            res.append(obj)
+    return res
diff --git a/pandas/core/indexes/api.py b/pandas/core/indexes/api.py
--- a/pandas/core/indexes/api.py
+++ b/pandas/core/indexes/api.py
@@ -23,8 +23,7 @@
            'PeriodIndex', 'DatetimeIndex',
            '_new_Index', 'NaT',
            '_ensure_index', '_get_na_value', '_get_combined_index',
-           '_get_objs_combined_axis',
-           '_get_distinct_indexes', '_union_indexes',
+           '_get_objs_combined_axis', '_union_indexes',
            '_get_consensus_names',
            '_all_indexes_same']
 
@@ -41,7 +40,7 @@ def _get_objs_combined_axis(objs, intersect=False, axis=0):
 
 def _get_combined_index(indexes, intersect=False):
     # TODO: handle index names!
-    indexes = _get_distinct_indexes(indexes)
+    indexes = com._get_distinct_objs(indexes)
     if len(indexes) == 0:
         return Index([])
     if len(indexes) == 1:
@@ -55,10 +54,6 @@ def _get_combined_index(indexes, intersect=False):
     return _ensure_index(union)
 
 
-def _get_distinct_indexes(indexes):
-    return list(dict((id(x), x) for x in indexes).values())
-
-
 def _union_indexes(indexes):
     if len(indexes) == 0:
         raise AssertionError('Must have at least 1 Index to union')


Test patch:
diff --git a/pandas/tests/reshape/test_concat.py b/pandas/tests/reshape/test_concat.py
--- a/pandas/tests/reshape/test_concat.py
+++ b/pandas/tests/reshape/test_concat.py
@@ -5,7 +5,7 @@
 from numpy.random import randn
 
 from datetime import datetime
-from pandas.compat import StringIO, iteritems
+from pandas.compat import StringIO, iteritems, PY2
 import pandas as pd
 from pandas import (DataFrame, concat,
                     read_csv, isna, Series, date_range,
@@ -1944,6 +1944,17 @@ def test_concat_categoricalindex(self):
                            index=exp_idx)
         tm.assert_frame_equal(result, exp)
 
+    def test_concat_order(self):
+        # GH 17344
+        dfs = [pd.DataFrame(index=range(3), columns=['a', 1, None])]
+        dfs += [pd.DataFrame(index=range(3), columns=[None, 1, 'a'])
+                for i in range(100)]
+        result = pd.concat(dfs).columns
+        expected = dfs[0].columns
+        if PY2:
+            expected = expected.sort_values()
+        tm.assert_index_equal(result, expected)
+
 
 @pytest.mark.parametrize('pdt', [pd.Series, pd.DataFrame, pd.Panel])
 @pytest.mark.parametrize('dt', np.sctypes['float'])