Repository: pandas-dev/pandas

Issue:
Dataframe rename issue.
I just upgrage from 0.11 to 0.12 version. And meet  dataframe rename error caused by upgrading.  (It worked well in 0.11) .   

```
>>> df4
                 TClose      RT    TExg
STK_ID RPT_Date                        
600809 20130331   22.02  0.0454  0.0422

>>> df5
                 STK_ID  RPT_Date STK_Name  TClose
STK_ID RPT_Date                                   
600809 20120930  600809  20120930     山西汾酒   38.05
       20121231  600809  20121231     山西汾酒   41.66
       20130331  600809  20130331     山西汾酒   30.01

>>> k=pd.merge(df4, df5, how='inner', left_index=True, right_index=True)
>>> k
                 TClose_x      RT    TExg  STK_ID  RPT_Date STK_Name  TClose_y
STK_ID RPT_Date                                                               
600809 20130331     22.02  0.0454  0.0422  600809  20130331     山西汾酒     30.01

>>> k.rename(columns={'TClose_x':'TClose', 'TClose_y':'QT_Close'})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "d:\Python27\lib\site-packages\pandas\core\base.py", line 40, in __repr__
    return str(self)
  File "d:\Python27\lib\site-packages\pandas\core\base.py", line 20, in __str__
    return self.__bytes__()
  File "d:\Python27\lib\site-packages\pandas\core\base.py", line 32, in __bytes__
    return self.__unicode__().encode(encoding, 'replace')
  File "d:\Python27\lib\site-packages\pandas\core\frame.py", line 668, in __unicode__
    self.to_string(buf=buf)
  File "d:\Python27\lib\site-packages\pandas\core\frame.py", line 1556, in to_string
    formatter.to_string()
  File "d:\Python27\lib\site-packages\pandas\core\format.py", line 294, in to_string
    strcols = self._to_str_columns()
  File "d:\Python27\lib\site-packages\pandas\core\format.py", line 239, in _to_str_columns
    str_columns = self._get_formatted_column_labels()
  File "d:\Python27\lib\site-packages\pandas\core\format.py", line 435, in _get_formatted_column_labels
    dtypes = self.frame.dtypes
  File "d:\Python27\lib\site-packages\pandas\core\frame.py", line 1696, in dtypes
    return self.apply(lambda x: x.dtype)
  File "d:\Python27\lib\site-packages\pandas\core\frame.py", line 4416, in apply
    return self._apply_standard(f, axis)
  File "d:\Python27\lib\site-packages\pandas\core\frame.py", line 4491, in _apply_standard
    raise e
TypeError: ("'NoneType' object is not iterable", u'occurred at index TExg')

>>> df4.dtypes
TClose    float64
RT        float64
TExg      float64
dtype: object

>>> df5.dtypes
STK_ID       object
RPT_Date     object
STK_Name     object
TClose      float64
dtype: object
>>> 
```



Discussion:
can you supply a reproducible for these initial frames (e.g. a function which does it exactly)

e.g. something that can be evaled to created it because need to reproduce the unicode characters
(this is a unicode error), just happens to show up in the dtype printing

`DataFrame([['foo',1.0....])`

i think that's a possibly spurious raise there...it should probably be a bare `raise` since `NoneType` not being iterable is not informative

i can repro this using the above frames

@halleygithub please supply some code to create the above frames.

there's a bug in `icol` or `BlockManager.iget`

ahh duplicate `TExg` block somehow...

we really need to remove that `raise e` there that's only way i was able to figure out this was in internals

no that raise is correct

just str(df)

huh? the raise doesn't show the correct location of the exception because it catches everything

here's part of the traceback

```
/home/phillip/Documents/code/py/pandas/pandas/core/frame.pyc in dtypes(self)
   1685     @property
   1686     def dtypes(self):
-> 1687         return self.apply(lambda x: x.dtype)
   1688
   1689     def convert_objects(self, convert_dates=True, convert_numeric=False, copy=True):

/home/phillip/Documents/code/py/pandas/pandas/core/frame.pyc in apply(self, func, axis, broadcast, raw, args, **kwds)
   4397                     return self._apply_raw(f, axis)
   4398                 else:
-> 4399                     return self._apply_standard(f, axis)
   4400             else:
   4401                 return self._apply_broadcast(f, axis)

/home/phillip/Documents/code/py/pandas/pandas/core/frame.pyc in _apply_standard(self, func, axis, ignore_failures)
   4472                     # no k defined yet
   4473                     pass
-> 4474                 raise e
   4475
   4476

TypeError: ("'NoneType' object is not iterable", u'occurred at index TExg')
```

this doesn't tell me anything about the location of the raise except that it was somewhere in looping thru `series_gen`

only when i removed `e` did the full traceback show up

maybe there's a way to show that without removing the `e`...

how would it be different anyway? would the possibly caught NameError / UnboundLocalError be raised instead?

```
In [4]: df4 = DataFrame({'TClose': [22.02], 'RT': [0.0454], 'TExg': [0.0422]}, index=MultiIndex.from_tuples([(600809, 20130331)], names=['STK_ID', 'RPT_Date']))

In [5]: df5 = DataFrame({'STK_ID': [600809] * 3, 'RPT_Date': [20120930,20121231,20130331], 'STK_Name': [u'饡驦', u'饡驦', u'饡驦'], 'TClose': [38.05, 41.66, 30.01]},index=MultiIndex.from_tuples([(600809, 20120930
), (600809, 20121231),(600809,20130331)], names=['STK_ID', 'RPT_Date']))

In [6]: k = merge(df4,df5,how='inner',left_index=True,right_index=True)
```

different characters but same error results.

curiously if you type `store k` then restart ipython, type `store -r k` and then

``` python
k.rename(columns={'TClose_x':'TClose'})
```

the error does _not_ show up :angry: 

I think there is a pr out there to take out the e

but regardless the apply hits the error but its really in the construction

can u post your creation example?

it's there

this seems fishy

```
ipdb> self.items
Index([u'RT', u'TClose', u'TExg', u'RPT_Date', u'STK_ID', u'STK_Name', u'TClose_y'], dtype=object)
ipdb> self.blocks
[ObjectBlock: [TExg], 1 x 1, dtype object, IntBlock: [RT, TClose], 2 x 1, dtype int64, FloatBlock: [RT, TClose, TExg, TClose_y], 4 x 1, dtype float64]
```

where is `RPT_Date` in the blocks?


Resolving patch:
diff --git a/pandas/tools/merge.py b/pandas/tools/merge.py
--- a/pandas/tools/merge.py
+++ b/pandas/tools/merge.py
@@ -683,6 +683,7 @@ def get_result(self):
         blockmaps = self._prepare_blocks()
         kinds = _get_merge_block_kinds(blockmaps)
 
+        result_is_unique = self.result_axes[0].is_unique
         result_blocks = []
 
         # maybe want to enable flexible copying <-- what did I mean?
@@ -692,6 +693,12 @@ def get_result(self):
                 if klass in mapping:
                     klass_blocks.extend((unit, b) for b in mapping[klass])
             res_blk = self._get_merged_block(klass_blocks)
+
+            # if we have a unique result index, need to clear the _ref_locs
+            # a non-unique is set as we are creating
+            if result_is_unique:
+                res_blk.set_ref_locs(None)
+
             result_blocks.append(res_blk)
 
         return BlockManager(result_blocks, self.result_axes)
@@ -1070,7 +1077,7 @@ def _concat_blocks(self, blocks):
             # map the column location to the block location
             # GH3602
             if not self.new_axes[0].is_unique:
-                block._ref_locs = indexer
+                block.set_ref_locs(indexer)
 
             return block
 


Test patch:
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -1,3 +1,5 @@
+# -*- coding: utf-8 -*-
+
 from __future__ import print_function
 # pylint: disable-msg=W0612,E1101
 from copy import deepcopy
@@ -2956,6 +2958,27 @@ def check(result, expected=None):
         expected = np.array([[1,2.5],[3,4.5]])
         self.assert_((result == expected).all().all())
 
+        # rename, GH 4403
+        df4 = DataFrame({'TClose': [22.02],
+                         'RT': [0.0454],
+                         'TExg': [0.0422]},
+                        index=MultiIndex.from_tuples([(600809, 20130331)], names=['STK_ID', 'RPT_Date']))
+
+        df5 = DataFrame({'STK_ID': [600809] * 3,
+                         'RPT_Date': [20120930,20121231,20130331],
+                         'STK_Name': [u('饡驦'), u('饡驦'), u('饡驦')],
+                         'TClose': [38.05, 41.66, 30.01]},
+                        index=MultiIndex.from_tuples([(600809, 20120930), (600809, 20121231),(600809,20130331)], names=['STK_ID', 'RPT_Date']))
+
+        k = pd.merge(df4,df5,how='inner',left_index=True,right_index=True)
+        result = k.rename(columns={'TClose_x':'TClose', 'TClose_y':'QT_Close'})
+        str(result)
+        result.dtypes
+
+        expected = DataFrame([[0.0454, 22.02, 0.0422, 20130331, 600809, u('饡驦'), 30.01 ]],
+                             columns=['RT','TClose','TExg','RPT_Date','STK_ID','STK_Name','QT_Close']).set_index(['STK_ID','RPT_Date'],drop=False)
+        assert_frame_equal(result,expected)
+
     def test_insert_benchmark(self):
         # from the vb_suite/frame_methods/frame_insert_columns
         N = 10