BUG: Possibly invalidate the item_cache when numpy implicty converts a v...
State: closed

closes #3970

`df['bb'].iloc[0] = 0.13`
here `bb` is put into the `_item_cache` and the first element assigned 0.13; `bb` is still a view onto the frame block

`df_tmp = df.iloc[[True]*len(df)]`
`df_tmp` is now a copy of df, but built up as a take block-by-block from `df`. I believe numpy then decides to invalidate the views to the memory in the float block (why I have no idea); `bb` in `df` is still holding the view in the `_item_cache` to the old memory location.

`df['bb'].iloc[0] = 0.15`
grabs `bb` from the cache and assigns 0.15 to it, BUT it is not longer a view onto the float block in `df` so nothing appears to get updated

I tried 2 fixes:

1) when getting an item from the `_item_cache` check if its a view of the underlying data - this works, but makes lookups way slow 
2) clear the cache when `taking` ON THE ORIGINAL frame, so future lookups will cache-miss and get the correct data from the block manager

Fundamentally we COULD do this in the first statement `df['bb'].iloc[0] = 0.13`, where we don't reset the cache, except we don't have a reference to df (well we do, but by the time the operation is carried out we have an operation on the series, so have lost the frame reference)

So this is fixed in this case, but not sure what other numpy ops implicity convert view-> copy (and we are holding onto the cached view).


Labels:

Comments:
@wesm this was pretty tricky....not 100% sure I am right, what do you think?

Where precisely was the type conversion happening before that was leading to a stale cached Series in `_item_cache`?

I don't think was any type conversion. As near as I can tell its somhow the `take`, doesn't make sense
If I test for a view e.g. ((`id(df['bb'].values.base) == id(df._data.blocks[0].values)`)  before and after this line:
`df_tmp = df.iloc[[True]*len(df)]`, its True before, and False afterwards...

even though I know we are not touching the memory on `df` at all (though it IS passed to a cython function for taking), so maybe it assumes we are touching it somehow?

FYI - there is a method on BlockManager to do this in this PR, `is_view`

@wesm any thoughts on this?

@wesm 
another thought here....

`df['bb'].iloc[0] = 0.15` is the fundamental issue here
the Frame has no idea that the object is being altered (which if it knew could invalidate its cache and solve the problem)

This is effectively copy-on-write, but in order to actually implement this a series would have to know that its being cached (trivial) - but the invalidation needs to occur whenever you are changing something in a series, but without storing a reference to the cacher (the dataframe), this is very hard I think

so just need to think about cases when you need to invalidate the cache in the frames

This PR solves the current issues, but not sure if there are other related issues lurking

@wesm ok..this solves the problem...keeping a weak ref to the cacher in the series (and invalidting when writing)...but take a took in any event

Can you please wait to merge this until I have time to look a bit closer? Will try to look today or tomorrow

np

This is more pernicious than I thought:

on master:

```
In [1]: paste
df = DataFrame({ "aa":range(5), "bb":[2.2]*5})
df["cc"] = 0.0
ck = [True]*len(df)
df["bb"].loc[0] = .13 # works
## -- End pasted text --

In [2]: df
Out[2]:
   aa    bb  cc
0   0  0.13   0
1   1  2.20   0
2   2  2.20   0
3   3  2.20   0
4   4  2.20   0

In [3]: df['bb'].iloc[0] = 0.17

In [4]: df
Out[4]:
   aa    bb  cc
0   0  0.13   0
1   1  2.20   0
2   2  2.20   0
3   3  2.20   0
4   4  2.20   0
```

going to dig in and figure out what in the hell is going on for my own sake

Block consolidation is not invalidating the cache:

```
In [19]: paste
df = DataFrame({ "aa":range(5), "bb":[2.2]*5})
df["cc"] = 0.0
ck = [True]*len(df)
print df._data
df["bb"].loc[0] = .13 # works
## -- End pasted text --
BlockManager
Items: Index([u'aa', u'bb', u'cc'], dtype=object)
Axis 1: Int64Index([0, 1, 2, 3, 4], dtype=int64)
FloatBlock: [bb], 1 x 5, dtype float64
IntBlock: [aa], 1 x 5, dtype int64
FloatBlock: [cc], 1 x 5, dtype float64

In [20]: df
Out[20]:
   aa    bb  cc
0   0  0.13   0
1   1  2.20   0
2   2  2.20   0
3   3  2.20   0
4   4  2.20   0

In [21]: df["bb"].iloc[0] = .17

In [22]: df
Out[22]:
   aa    bb  cc
0   0  0.13   0
1   1  2.20   0
2   2  2.20   0
3   3  2.20   0
4   4  2.20   0

In [23]: df._data
Out[23]:
BlockManager
Items: Index([u'aa', u'bb', u'cc'], dtype=object)
Axis 1: Int64Index([0, 1, 2, 3, 4], dtype=int64)
FloatBlock: [bb, cc], 2 x 5, dtype float64
IntBlock: [aa], 1 x 5, dtype int64
```

Fix coming shortly

yep looks right....

closing in favor of #4077