Repository: pandas-dev/pandas

Issue:
BUG: sort_index/sortlevel fails  MultiIndex after columns are added.
I have a `DataFrame` with a `MultiIndex` on the columns.  The first level of the MultiIndex contains `str`ings.  The second, `float`s (though the problem persists if the second level is `int`s).  I add a column to the `DataFrame` (which should _not_ come last if the columns are sorted).  I try to sort the `DataFrame`.  The result does not seem to be sorted.  The behavior is fine if the columns are simply an `Index` (even after adding columns).  And the sort works fine in the `MultiIndex` case as long as no columns have been added since the `DataFrame` was created.

MWE:

```
import pandas as pd
import numpy as np

np.random.seed(0)
data = np.random.randn(3,4)

df_multi_float = pd.DataFrame(data, index=list('def'), columns=pd.MultiIndex.from_tuples([('red', i) for i in [1., 3., 2., 5.]]))

print df_multi_float

#OUTPUT
        red                              
          1         3         2         5
d  1.764052  0.400157  0.978738  2.240893
e  1.867558 -0.977278  0.950088 -0.151357
f -0.103219  0.410599  0.144044  1.454274
```

This sorts just fine as it isnow:

```
print df_multi_float.sort_index(axis=1)

#OUTPUT
        red                              
          1         2         3         5
d  1.764052  0.978738  0.400157  2.240893
e  1.867558  0.950088 -0.977278 -0.151357
f -0.103219  0.144044  0.410599  1.454274
```

But if I add columns to both this `DataFrame and then show it sorted, I get what looks to be a wrong result (the new column remains last, rather than being placed second-to-last as it should be):

```
df_multi_float[('red', 4.0)] = 'world'

print df_multi_float.sort_index(axis=1)

#OUTPUT
        red                                  red
          1         2         3         5      4
d  1.764052  0.978738  0.400157  2.240893  world
e  1.867558  0.950088 -0.977278 -0.151357  world
f -0.103219  0.144044  0.410599  1.454274  world
```

I'm able to produce this behavior on two systems.  The first runs Pandas 0.14.0 and Numpy 1.8.1 and the second runs Pandas 0.14.1 and Numpy 1.8.2.  This issue is described here: http://stackoverflow.com/questions/25287130/pandas-sort-index-fails-with-multiindex-containing-floats-as-one-level-when-col?noredirect=1#comment39408150_25287130



Discussion:
@8one6 

just make the example ONLY the muli_float and just run it on one. simplier/shorter is much better

Sure.  I'll make the change above.  I had thought it was important to show that it worked fine with a float index that was not a multiindex.  But no problem.  (Done.)

@8one6 thanks, your title says it all though

@jreback Just one note on your title update.  The problem is still there if all of the `float`s in my example are replaced by `int`s.

yeh I think has to do with the adding, will look

I think the fix for this will be much deeper in the bowels of Pandas indices than I'm able to handle.  Is there any other way I could help toward a patch for this bug?

np. would appreciate a pull-request on any other issue. thanks!

In an attempt to get around this issue, I started sticking a character at the end of some of my column names (to turn them from numbers into strings).  While the `sort_index` function seems to work fine when the column names are purely letters, this bug seems to come back when they are a mix of numerals and letters.

```
import pandas as pd
import numpy as np

np.random.seed(0)
data = np.random.randn(3, 4)
df_all_string_indices = pd.DataFrame(data,
                                   columns=pd.MultiIndex.from_product([['2b', '1b'], ['2c', '1c']], names=['one', 'two']),
                                   index=list('xyz'))

print df_all_string_indices

one        2b                  1b          
two        2c        1c        2c        1c
x    1.764052  0.400157  0.978738  2.240893
y    1.867558 -0.977278  0.950088 -0.151357
z   -0.103219  0.410599  0.144044  1.454274
```

which looks fine.  and sorts fine:

```
print df_all_string_indices.sort_index(axis=1)

one        1b                  2b          
two        1c        2c        1c        2c
x    2.240893  0.978738  0.400157  1.764052
y   -0.151357  0.950088 -0.977278  1.867558
z    1.454274  0.144044  0.410599 -0.103219
```

But when I add a new column and try to sort, this still goes wrong:

```
df_all_string_indices[('1a', 'new')] =  'hello'
print df_all_string_indices.sort_index(axis=1)

one        1b                  2b               1a
two        1c        2c        1c        2c    new
x    2.240893  0.978738  0.400157  1.764052  hello
y   -0.151357  0.950088 -0.977278  1.867558  hello
z    1.454274  0.144044  0.410599 -0.103219  hello
```

(I.e. I think that after the `sort_index`, the `('1a', 'new')` column should come first.)


Resolving patch:
diff --git a/pandas/core/format.py b/pandas/core/format.py
--- a/pandas/core/format.py
+++ b/pandas/core/format.py
@@ -625,10 +625,17 @@ def is_numeric_dtype(dtype):
             fmt_columns = columns.format(sparsify=False, adjoin=False)
             fmt_columns = lzip(*fmt_columns)
             dtypes = self.frame.dtypes.values
+
+            # if we have a Float level, they don't use leading space at all
+            restrict_formatting = any([ l.is_floating for l in columns.levels ])
             need_leadsp = dict(zip(fmt_columns, map(is_numeric_dtype, dtypes)))
-            str_columns = list(zip(*[
-                [' ' + y if y not in self.formatters and need_leadsp[x]
-                 else y for y in x] for x in fmt_columns]))
+
+            def space_format(x,y):
+                if y not in self.formatters and need_leadsp[x] and not restrict_formatting:
+                    return ' ' + y
+                return y
+
+            str_columns = list(zip(*[ [ space_format(x,y) for y in x ] for x in fmt_columns ]))
             if self.sparsify:
                 str_columns = _sparsify(str_columns)
 
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -2770,6 +2770,12 @@ def trans(v):
                                     na_position=na_position)
 
         elif isinstance(labels, MultiIndex):
+
+            # make sure that the axis is lexsorted to start
+            # if not we need to reconstruct to get the correct indexer
+            if not labels.is_lexsorted():
+                labels = MultiIndex.from_tuples(labels.values)
+
             indexer = _lexsort_indexer(labels.labels, orders=ascending,
                                        na_position=na_position)
             indexer = com._ensure_platform_int(indexer)
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -1628,6 +1628,7 @@ def sort_index(self, axis=0, ascending=True):
 
         new_axis = labels.take(sort_index)
         return self.reindex(**{axis_name: new_axis})
+
     _shared_docs['reindex'] = """
         Conform %(klass)s to new index with optional filling logic, placing
         NA/NaN in locations having no value in the previous index. A new object
@@ -3558,10 +3559,10 @@ def _tz_convert(ax, tz):
         result = self._constructor(self._data, copy=copy)
         result.set_axis(axis,ax)
         return result.__finalize__(self)
-    
+
     @deprecate_kwarg(old_arg_name='infer_dst', new_arg_name='ambiguous',
                      mapping={True: 'infer', False: 'raise'})
-    def tz_localize(self, tz, axis=0, level=None, copy=True, 
+    def tz_localize(self, tz, axis=0, level=None, copy=True,
                     ambiguous='raise'):
         """
         Localize tz-naive TimeSeries to target time zone
@@ -3583,7 +3584,7 @@ def tz_localize(self, tz, axis=0, level=None, copy=True,
             - 'raise' will raise an AmbiguousTimeError if there are ambiguous times
         infer_dst : boolean, default False (DEPRECATED)
             Attempt to infer fall dst-transition hours based on order
-            
+
         Returns
         -------
         """


Test patch:
diff --git a/pandas/tests/test_multilevel.py b/pandas/tests/test_multilevel.py
--- a/pandas/tests/test_multilevel.py
+++ b/pandas/tests/test_multilevel.py
@@ -214,6 +214,44 @@ def test_sort_index_preserve_levels(self):
         result = self.frame.sort_index()
         self.assertEqual(result.index.names, self.frame.index.names)
 
+    def test_sorting_repr_8017(self):
+
+        np.random.seed(0)
+        data = np.random.randn(3,4)
+
+        for gen, extra in [([1.,3.,2.,5.],4.),
+                           ([1,3,2,5],4),
+                           ([Timestamp('20130101'),Timestamp('20130103'),Timestamp('20130102'),Timestamp('20130105')],Timestamp('20130104')),
+                           (['1one','3one','2one','5one'],'4one')]:
+            columns = MultiIndex.from_tuples([('red', i) for i in gen])
+            df = DataFrame(data, index=list('def'), columns=columns)
+            df2 = pd.concat([df,DataFrame('world',
+                                          index=list('def'),
+                                          columns=MultiIndex.from_tuples([('red', extra)]))],axis=1)
+
+            # check that the repr is good
+            # make sure that we have a correct sparsified repr
+            # e.g. only 1 header of read
+            self.assertEqual(str(df2).splitlines()[0].split(),['red'])
+
+            # GH 8017
+            # sorting fails after columns added
+
+            # construct single-dtype then sort
+            result = df.copy().sort_index(axis=1)
+            expected = df.iloc[:,[0,2,1,3]]
+            assert_frame_equal(result, expected)
+
+            result = df2.sort_index(axis=1)
+            expected = df2.iloc[:,[0,2,1,4,3]]
+            assert_frame_equal(result, expected)
+
+            # setitem then sort
+            result = df.copy()
+            result[('red',extra)] = 'world'
+            result = result.sort_index(axis=1)
+            assert_frame_equal(result, expected)
+
     def test_repr_to_string(self):
         repr(self.frame)
         repr(self.ymd)