Repository: pandas-dev/pandas

Issue:
view into modified dataframe of ints causes subsequent set_value to not work properly
The following code snippet produces unexpected output:

``` python
import pandas as pd
import numpy as np
x = pd.DataFrame(data=np.zeros((5,5),dtype=np.int),columns=['a','b','c','d','e'])
x['f'] = 0
x.f.values[3]=1
y = x.iloc[np.arange(2,len(x))]
x.f.values[3]=2
print x
print x.f.values
```

```
Out[130]: 
   a  b  c  d  e  f
0  0  0  0  0  0  0
1  0  0  0  0  0  0
2  0  0  0  0  0  0
3  0  0  0  0  0  1
4  0  0  0  0  0  0
[0 0 0 2 0]
```

The expected output is:

```
Out[131]: 
   a  b  c  d  e  f
0  0  0  0  0  0  0
1  0  0  0  0  0  0
2  0  0  0  0  0  0
3  0  0  0  0  0  2
4  0  0  0  0  0  0
[0 0 0 2 0]
```

The behavior only manifests if the dataframe has had a column added to it (f in this case), if the columns are ints, and if the view is constructed with certain kinds of indexing (y=x.iloc[2:] works fine for instance).

Here is my version info:
## INSTALLED VERSIONS

commit: None
python: 2.7.7.final.0
python-bits: 64
OS: Linux
OS-release: 3.10.0-229.4.2.el7.jump.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.15.2.dev
nose: 1.3.4
Cython: 0.20.2
numpy: 1.9.0
scipy: 0.14.1rc1.dev-Unknown
statsmodels: None
IPython: 3.1.0
sphinx: 1.3b1
patsy: 0.3.0
dateutil: 2.2
pytz: 2014.10
bottleneck: 0.8.0
tables: 3.1.1
numexpr: 2.3.1
matplotlib: 1.4.0
openpyxl: 2.1.4
xlrd: None
xlwt: None
xlsxwriter: None
lxml: 3.4.2
bs4: 4.3.2
html5lib: None
httplib2: None
apiclient: None
rpy2: 2.4.3
sqlalchemy: None
pymysql: 0.6.2.None
psycopg2: 2.5.4 (dt dec pq3 ext)



Discussion:
I can reproduce this on master. Definitely a bug -- thanks for the report!

I know this looks like a bug, and might be. However, you are violating the guarantees. The `.values` objects does not guarantee that you will get a referenceable consolidated view for a `DataFrame`. 

For a `Series` this is always True, and _generally_ it is true for a `DataFrame` when looking at a single dtype (but this is NEVER true for multi-dtype selection as you will always get a copy), but for a single dtype view (or a view on a multi-dtype that only is a single dtype).

This implementation is lazy about consolidation (e.g. it won't actually consolidate on a single insert like this, but will later on).

So we could call it a bug if it should _always_ consolidate on inserts (but this _can_ severly impact performance, that's why its like this). Or say that what you are doing (e.g. by ASSIGNING to `.values` directly), is technically not allowed (and in any event its not good practice ever).

```
In [34]: x = pd.DataFrame(data=np.zeros((5,5),dtype=np.int),columns=['a','b','c','d','e'])

In [35]: x._data
Out[35]: 
BlockManager
Items: Index([u'a', u'b', u'c', u'd', u'e'], dtype='object')
Axis 1: Int64Index([0, 1, 2, 3, 4], dtype='int64')
IntBlock: slice(0, 5, 1), 5 x 5, dtype: int64

In [36]: x['f'] = 0

In [37]: x._data
Out[37]: 
BlockManager
Items: Index([u'a', u'b', u'c', u'd', u'e', u'f'], dtype='object')
Axis 1: Int64Index([0, 1, 2, 3, 4], dtype='int64')
IntBlock: slice(0, 5, 1), 5 x 5, dtype: int64
IntBlock: slice(5, 6, 1), 1 x 5, dtype: int64

In [38]: x.f.values.base
Out[38]: array([0, 0, 0, 0, 0])

In [39]: x.e.values.base
Out[39]: 
array([[0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0]])
In [41]: x._consolidate_inplace()

In [42]: x._data
Out[42]: 
BlockManager
Items: Index([u'a', u'b', u'c', u'd', u'e', u'f'], dtype='object')
Axis 1: Int64Index([0, 1, 2, 3, 4], dtype='int64')
IntBlock: slice(0, 6, 1), 6 x 5, dtype: int64

In [43]: x.values.base
Out[43]: 
array([[0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0]])
```

@jreback the bug (IMO) is that `x` and `x.f` are inconsistent. Assigning to `.values` is a bit of a side point.

@shoyer you'd have to prove it. How are they inconsistent?

@jreback  compare lines 8 and 9 below:

```

In [2]: import numpy as np

In [3]: x = pd.DataFrame(data=np.zeros((5,5),dtype=np.int),columns=['a','b','c','d','e'])

In [4]: x['f'] = 0

In [5]: x.f.values[3]=1

In [6]: y = x.iloc[np.arange(2,len(x))]

In [7]: x.f.values[3]=2

In [8]: x
Out[8]:
   a  b  c  d  e  f
0  0  0  0  0  0  0
1  0  0  0  0  0  0
2  0  0  0  0  0  0
3  0  0  0  0  0  1
4  0  0  0  0  0  0

In [9]: x.f
Out[9]:
0    0
1    0
2    0
3    2
4    0
Name: f, dtype: int64
```

you are using `.values`, which violates guarantees

```
In [27]: x = pd.DataFrame(data=np.zeros((5,5),dtype=np.int),columns=['a','b','c','d','e'])

In [28]: x['f'] = 0

In [29]: x.ix[3,'f'] = 2

In [30]: x
Out[30]: 
   a  b  c  d  e  f
0  0  0  0  0  0  0
1  0  0  0  0  0  0
2  0  0  0  0  0  0
3  0  0  0  0  0  2
4  0  0  0  0  0  0

In [31]: x['f']
Out[31]: 
0    0
1    0
2    0
3    2
4    0
Name: f, dtype: int64
```

> you are using `.values`, which violates guarantees

It should not be possible to put a dataframe into an inconsistent state, even if you're mis-using the pandas API.

The state IS consistent. The issue is that the user is holding onto an incorrect reference. This is exactly `SettingWithCopyError`.

From the [docs](http://pandas.pydata.org/pandas-docs/version/0.16.1/basics.html#attributes-and-the-raw-ndarray-s)

> If a DataFrame or Panel contains homogeneously-typed data, the ndarray can actually be modified in-place, and the changes will be reflected in the data structure.

So we'll need to update that or call this a bug. Perhaps we should remove that section either way and encourage users to use `.loc` for setting?

@jreback what is the incorrect reference? Do we cache `x.f` instead of computing it on demand?


Resolving patch:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -1352,6 +1352,7 @@ def take(self, indices, axis=0, convert=True, is_copy=True):
         taken : type of caller
         """
 
+        self._consolidate_inplace()
         new_data = self._data.take(indices,
                                    axis=self._get_block_manager_axis(axis),
                                    convert=True, verify=True)
@@ -2128,8 +2129,10 @@ def _protect_consolidate(self, f):
         return result
 
     def _consolidate_inplace(self):
-        f = lambda: self._data.consolidate()
-        self._data = self._protect_consolidate(f)
+        """ we are inplace consolidating; return None """
+        def f():
+            self._data = self._data.consolidate()
+        self._protect_consolidate(f)
 
     def consolidate(self, inplace=False):
         """
diff --git a/pandas/core/indexing.py b/pandas/core/indexing.py
--- a/pandas/core/indexing.py
+++ b/pandas/core/indexing.py
@@ -369,6 +369,7 @@ def _setitem_with_indexer(self, indexer, value):
                     # we can directly set the series here
                     # as we select a slice indexer on the mi
                     idx = index._convert_slice_indexer(idx)
+                    obj._consolidate_inplace()
                     obj = obj.copy()
                     obj._data = obj._data.setitem(indexer=tuple([idx]), value=value)
                     self.obj[item] = obj
@@ -396,6 +397,7 @@ def setter(item, v):
                     s = v
                 else:
                     # set the item, possibly having a dtype change
+                    s._consolidate_inplace()
                     s = s.copy()
                     s._data = s._data.setitem(indexer=pi, value=v)
                     s._maybe_update_cacher(clear=True)
@@ -492,6 +494,7 @@ def can_do_equal_len():
             self.obj._check_is_chained_assignment_possible()
 
             # actually do the set
+            self.obj._consolidate_inplace()
             self.obj._data = self.obj._data.setitem(indexer=indexer, value=value)
             self.obj._maybe_update_cacher(clear=True)
 
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -2830,13 +2830,13 @@ def consolidate(self):
             return self
 
         bm = self.__class__(self.blocks, self.axes)
+        bm._is_consolidated = False
         bm._consolidate_inplace()
         return bm
 
     def _consolidate_inplace(self):
         if not self.is_consolidated():
             self.blocks = tuple(_consolidate(self.blocks))
-
             self._is_consolidated = True
             self._known_consolidated = True
             self._rebuild_blknos_and_blklocs()


Test patch:
diff --git a/pandas/tests/test_indexing.py b/pandas/tests/test_indexing.py
--- a/pandas/tests/test_indexing.py
+++ b/pandas/tests/test_indexing.py
@@ -3527,6 +3527,18 @@ def test_cache_updating(self):
         result = df.loc[(0,0),'z']
         self.assertEqual(result, 2)
 
+        # 10264
+        df = DataFrame(np.zeros((5,5),dtype='int64'),columns=['a','b','c','d','e'],index=range(5))
+        df['f'] = 0
+        df.f.values[3] = 1
+        y = df.iloc[np.arange(2,len(df))]
+        df.f.values[3] = 2
+        expected = DataFrame(np.zeros((5,6),dtype='int64'),columns=['a','b','c','d','e','f'],index=range(5))
+        expected.at[3,'f'] = 2
+        assert_frame_equal(df, expected)
+        expected = Series([0,0,0,2,0],name='f')
+        assert_series_equal(df.f, expected)
+
     def test_slice_consolidate_invalidate_item_cache(self):
 
         # this is chained assignment, but will 'work'