Repository: pandas-dev/pandas

Issue:
BUG: sorting with tuples and Multi-Index creation
Hello,

I'm using python 2.7.9 and pandas 0.15.2.
I'm trying to concatenate two DataFrames (to_concat is a list of 2 DataFrames) and I have the following error.

```
  File "/Users/alexisbenoist/Documents/CoTH/AFDS/app/core.py", line 67, in train_dataflow
    df = pd.concat(to_concat)
  File "/Users/alexisbenoist/.virtualenvs/afds/lib/python2.7/site-packages/pandas/tools/merge.py", line 725, in concat
    return op.get_result()
  File "/Users/alexisbenoist/.virtualenvs/afds/lib/python2.7/site-packages/pandas/tools/merge.py", line 896, in get_result
    mgrs_indexers, self.new_axes, concat_axis=self.axis, copy=self.copy)
  File "/Users/alexisbenoist/.virtualenvs/afds/lib/python2.7/site-packages/pandas/core/internals.py", line 4046, in concatenate_block_managers
    for placement, join_units in concat_plan]
  File "/Users/alexisbenoist/.virtualenvs/afds/lib/python2.7/site-packages/pandas/core/internals.py", line 4135, in concatenate_join_units
    empty_dtype, upcasted_na = get_empty_dtype_and_na(join_units)
  File "/Users/alexisbenoist/.virtualenvs/afds/lib/python2.7/site-packages/pandas/core/internals.py", line 4124, in get_empty_dtype_and_na
    raise AssertionError("invalid dtype determination in get_concat_dtype")
AssertionError: invalid dtype determination in get_concat_dtype
```

I put the pickle of the DataFrames in a [gist](https://gist.github.com/Alexis-benoist/ba7f38deccc70d46c32f) using (I can upload it somewhere else if it's more convenient).

```
import pickle
with open(path, 'w+') as f:
    pickle.dump(to_concat, f)
```

I feel that I'm doing something wrong or it's a bug from pandas because if do: 

```
pd.DataFrame(np.concatenate(to_concat), columns=to_concat[0].columns.tolist())
```

It gives a result which seems logical.

Any input is welcome.

Cheers,
Alexis.



Discussion:
Do you have a small script to generate two DataFrames that cause the error? That would be preferable to a pickle.
Also show exactly how you calling `concat`.

The script which generates the dataframes is a bit complicated but I can provide csv exports [df1](https://gist.github.com/Alexis-benoist/1a36c69a818a984c2950) [df2](https://gist.github.com/Alexis-benoist/d5aae92d79eaecd70789).

I tried to reproduce the bug using, the csv and the dataframe but unfortunately it works.  I also tried to serialize the array using np.save, but it also works.

``` python
import pandas as pd
from ast import literal_eval as literal_eval_base

def literal_eval(s):
    try:
        return literal_eval_base(s)
    except Exception as e:
        return s


df_1 = pd.read_csv('df_1.csv')
df_2 = pd.read_csv('df_2.csv')

df_1.rename(columns=literal_eval, inplace=True)
df_2.rename(columns=literal_eval, inplace=True)
to_concat = [df_1, df_2]
pd.concat(to_concat)
```

I think that the bug is on the line 4039 of core/internals in the combine_concat_plans func.
Because, we can see below that the value in the iterator (JoinUnit(None, {}), JoinUnit(None, {})) is not the same as the one in the evaluated list (None of them is None).  
<img width="957" alt="capture d ecran 2015-07-29 a 16 05 57" src="https://cloud.githubusercontent.com/assets/4437159/8959549/c27127fe-360b-11e5-80c2-997d556e1727.png">

I don't really know how i could send you the DataFrame in a readable format for you. 

I'm wondering, what could be the difference between the csv and the pandas (appart from the unamed column). 

The get_mgr_concatenation_plan does not seems coherent as we can see on the picture below. When it's called by the code  (the length is 1) but when I call it from the debugger, the length is 6.

<img width="1398" alt="capture d ecran 2015-07-29 a 16 59 58" src="https://cloud.githubusercontent.com/assets/4437159/8961145/cf7fedce-3613-11e5-987c-10d057ae6018.png">

:+1: 

This was a bit tricky to repro. However, the bottom line is their is a small edge case bug that you hit, though you should NOT ever use tuples in an index, unless it really is a multi-index, as these are by-definition the markers for multi-index guessing.

This is very odd in how things are sorted, not sure why exactly python does this (this is py2).

```
In [23]: sorted([u'class',(u'internal',u'__delete__')])
Out[23]: [(u'internal', u'__delete__'), u'class']

In [24]: sorted(['class',('internal','__delete__')])
Out[24]: ['class', ('internal', '__delete__')]
```

So this makes a determination if the guessed index is a multi-index or not (this is the bug part, I think we can be more strict, IOW we are only checking the first element if its a tuple; if it is we try to create a multi-index).

```
In [29]: Index([('A',1),'B'])
In [29]: Index([('A',1),'B'])
Out[29]: 
MultiIndex(levels=[[u'A', u'B'], [1]],
           labels=[[0, 1], [0, -1]])

In [30]: Index(['B',('A',1)])
Out[30]: Index([u'B', (u'A', 1)], dtype='object')
```

Here's a repro example. Note that if these are strings (e.g. no `u`), then this would WORK!.

```
In [26]: df1 = DataFrame({ u'A' : 'foo', (u'B',1) : 'bar' },index=range(2))

In [27]: df2 = DataFrame({ u'B' : 'foo', (u'B',1) : 'bar' },index=range(2))

In [28]: pd.concat([df1,df2])
AssertionError: invalid dtype determination in get_concat_dtype
```


Resolving patch:
diff --git a/pandas/core/index.py b/pandas/core/index.py
--- a/pandas/core/index.py
+++ b/pandas/core/index.py
@@ -164,16 +164,21 @@ def __new__(cls, data=None, dtype=None, copy=False, name=None, fastpath=False,
             cls._scalar_data_error(data)
         else:
             if tupleize_cols and isinstance(data, list) and data and isinstance(data[0], tuple):
-                try:
 
-                    # must be orderable in py3
-                    if compat.PY3:
-                        sorted(data)
-                    return MultiIndex.from_tuples(
-                        data, names=name or kwargs.get('names'))
-                except (TypeError, KeyError):
-                    # python2 - MultiIndex fails on mixed types
-                    pass
+                # we must be all tuples, otherwise don't construct
+                # 10697
+                if all( isinstance(e, tuple) for e in data ):
+
+                    try:
+
+                        # must be orderable in py3
+                        if compat.PY3:
+                            sorted(data)
+                        return MultiIndex.from_tuples(
+                            data, names=name or kwargs.get('names'))
+                    except (TypeError, KeyError):
+                        # python2 - MultiIndex fails on mixed types
+                        pass
 
             # other iterable of some kind
             subarr = com._asarray_tuplesafe(data, dtype=object)


Test patch:
diff --git a/pandas/tests/test_index.py b/pandas/tests/test_index.py
--- a/pandas/tests/test_index.py
+++ b/pandas/tests/test_index.py
@@ -553,6 +553,15 @@ def test_constructor_corner(self):
         # corner case
         self.assertRaises(TypeError, Index, 0)
 
+    def test_consruction_list_mixed_tuples(self):
+        # 10697
+        # if we are constructing from a mixed list of tuples, make sure that we
+        # are independent of the sorting order
+        idx1 = Index([('A',1),'B'])
+        self.assertIsInstance(idx1, Index) and self.assertNotInstance(idx1, MultiIndex)
+        idx2 = Index(['B',('A',1)])
+        self.assertIsInstance(idx2, Index) and self.assertNotInstance(idx2, MultiIndex)
+
     def test_constructor_from_series(self):
 
         expected = DatetimeIndex([Timestamp('20110101'),Timestamp('20120101'),Timestamp('20130101')])
diff --git a/pandas/tools/tests/test_merge.py b/pandas/tools/tests/test_merge.py
--- a/pandas/tools/tests/test_merge.py
+++ b/pandas/tools/tests/test_merge.py
@@ -2002,6 +2002,13 @@ def test_dups_index(self):
         result = df.append(df)
         assert_frame_equal(result, expected)
 
+    def test_with_mixed_tuples(self):
+        # 10697
+        # columns have mixed tuples, so handle properly
+        df1 = DataFrame({ u'A' : 'foo', (u'B',1) : 'bar' },index=range(2))
+        df2 = DataFrame({ u'B' : 'foo', (u'B',1) : 'bar' },index=range(2))
+        result = concat([df1,df2])
+
     def test_join_dups(self):
 
         # joining dups