Repository: pandas-dev/pandas

Issue:
 Some items were not contained in blocks AssertionError
The code below causes the DataFrame internals to get confused and throws an AssertionError on `print df.values`. The code looks contrived, but its just a simplified version of something I was trying to do. It doesn't seem to matter whether any columns are actually renamed so I simplified that also, but the rename calls do contribute to the problem.

```
import pandas as pd
df = pd.DataFrame({'a': [1, 2],
                   'b': [3, 4],
                   'c': [5, 6]})

df = df.set_index(['a', 'b'])
df = df.unstack()
df = df.rename(columns={'a': 'd'})
df = df.reset_index()
print df._data
df = df.rename(columns={})
print df._data
print df.values

```

```
BlockManager
Items: MultiIndex
[(u'a', u''), (u'c', 3), (u'c', 4)]
Axis 1: Int64Index([0, 1], dtype=int64)
FloatBlock: [(c, 3), (c, 4)], 2 x 2, dtype float64
IntBlock: [(a, )], 1 x 2, dtype int64

BlockManager
Items: MultiIndex
[(u'a', u''), (u'c', 3), (u'c', 4)]
Axis 1: Int64Index([0, 1], dtype=int64)
FloatBlock: [(a, ), (c, 3)], 2 x 2, dtype float64
IntBlock: [(a, )], 1 x 2, dtype int64

Traceback (most recent call last):
  File "/home/mtk/bug.py", line 13, in <module>
    print df.values
  File "/Users/mtk/Source/pandas/pandas/core/frame.py", line 1779, in as_matrix
    return self._data.as_matrix(columns).T
  File "/Users/mtk/Source/pandas/pandas/core/internals.py", line 1513, in as_matrix
    mat = self._interleave(self.items)
  File "/Users/mtk/Source/pandas/pandas/core/internals.py", line 1549, in _interleave
    raise AssertionError('Some items were not contained in blocks')
AssertionError: Some items were not contained in blocks

```

```
>>> pd.__version__
'0.11.1.dev-8a242d2'
```



Discussion:
the 2nd rename should not be allowed and is prob causing an issue (eg state gets messed up)

I think `rename({})` should be allowed, as it could be you use  a generated dictionary (which could be empty) to rename with. But agree that's probably where the issue lies.

Here's a simpler snippet that reproduces the problem:

```
import pandas as pd
df = pd.DataFrame({'b': [1.1, 2.2]})
df = df.rename(columns={})
df.insert(0, 'a', [1, 2])
df = df.rename(columns={})
print df.values
```

To be clear, this has nothing to do with passing an empty dict to rename. I know a little about the internals now, but not enough fix this one. I believe the bug is in insert. The first rename just happens to cause ref_locs to be calculated on the blocks. The insert method fails to update these ref_locs when a new columns is inserted (block.ref_items is updated but block.ref_locs is not). The second rename then causes block.items to be calculated with incorrect ref_locs. Things go downhill from there.

This is a regression from 11.0.


Resolving patch:
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -85,6 +85,8 @@ def set_ref_items(self, ref_items, maybe_rename=True):
         """
         if not isinstance(ref_items, Index):
             raise AssertionError('block ref_items must be an Index')
+        if maybe_rename == 'clear':
+            self._ref_locs = None
         if maybe_rename:
             self.items = ref_items.take(self.ref_locs)
         self.ref_items = ref_items
@@ -1798,12 +1800,18 @@ def insert(self, loc, item, value, allow_duplicates=False):
 
         if len(self.blocks) > 100:
             self._consolidate_inplace()
+        elif new_items.is_unique:
+            self.set_items_clear(new_items)
 
         self._known_consolidated = False
 
     def set_items_norename(self, value):
         self.set_axis(0, value, maybe_rename=False, check_axis=False)
 
+    def set_items_clear(self, value):
+        """ clear the ref_locs on all blocks """
+        self.set_axis(0, value, maybe_rename='clear', check_axis=False)
+
     def _delete_from_all_blocks(self, loc, item):
         """ delete from the items loc the item
             the item could be in multiple blocks which could
@@ -1914,7 +1922,7 @@ def _add_new_block(self, item, value, loc=None):
             # and reset
             self._reset_ref_locs()
             self._set_ref_locs(do_refs=True)
-            
+
     def _find_block(self, item):
         self._check_have(item)
         for i, block in enumerate(self.blocks):


Test patch:
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -10018,6 +10018,21 @@ def test_columns_with_dups(self):
         xp.columns = ['A', 'A', 'B']
         assert_frame_equal(rs, xp)
 
+    def test_insert_column_bug_4032(self):
+        # GH4032, inserting a column and renaming causing errors
+        df = DataFrame({'b': [1.1, 2.2]})
+        df = df.rename(columns={})
+        df.insert(0, 'a', [1, 2])
+
+        result = df.rename(columns={})
+        expected = DataFrame([[1,1.1],[2, 2.2]],columns=['a','b'])
+        assert_frame_equal(result,expected)
+        df.insert(0, 'c', [1.3, 2.3])
+
+        result = df.rename(columns={})
+        expected = DataFrame([[1.3,1,1.1],[2.3,2, 2.2]],columns=['c','a','b'])
+        assert_frame_equal(result,expected)
+
     def test_cast_internals(self):
         casted = DataFrame(self.frame._data, dtype=int)
         expected = DataFrame(self.frame._series, dtype=int)