Repository: pandas-dev/pandas

Issue:
Improve clarity around when SettingWithCopyWarning can be ignored (if ever?)
I [Edit: thought I got] a `SettingWarningCopyWarning` when running the following code:

``` python
frame[columnone][frame[columntwo]>x]=y
```

The docs seem to imply that I can safely ignore this error.

``` python
# passed via reference (will stay)
In [273]: dfb['c'][dfb.a.str.startswith('o')] = 42
```

Is it therefore generally true that I can always ignore errors when the command I'm running is of the form:

``` python
frame[columnames][booleanconditiononframe]=x
```

I use this all the time, and I _think_ it always works. If that's true, it would be great if it was specifically called out in a 'you can ignore this warning when...' section. If that's not true (e.g. it works for scalar x / numeric x but not for series x), it would be great if that were called out too.

Sorry if this seems trivial, but I'm trying to explain this to a colleague who's new to pandas and he's confused, and it looks like even old hands can be confused when confronted with this warning (see #6757).

[Edit: looking at his code again, I suspect the SettingWithCopy warning was from a different line of code - (I wish warnings made it clear where they were from). All the same, it would still be great if the docs could be clear if there are circumstances where you always get a reference rather than a copy]



Discussion:
Their are no cases that I am aware that you should actually ignore this warning. Your code _may_ still work, that's why its a warning. It has to do with whether what you are modifying is actually a copy of a view. In general if its a single dtyped frame it _mostly_ will work, BUT NOT ALWAYS. And that's the rub. If you do certain types of indexing it will never work, others it will work. You are really playing with fire. 

This is kept on as an actual error in the entire pandas test suite. You should really never do this.

Use 

`df.loc[row_indexer,col_indexer] = value` and it will ALWAYS work, no matter what you are doing.

Otherwise you are just holding a gun, maybe shooting yourself when you least expect it.

If you think the docs are unclear, pls submit a pull-request to clarify. It is important from a user's perspective that they are very clear.

That example shows a case where chained indexing DOES WORK, but does not imply that you should use it. (again if its unclear from reading, pls lmk / submit a PR to fix the docs).

The reason you would ignore the warning is that AFAIK pandas can't tell the difference between

``` python
# WRONG
frame[columnone][frame[columntwo]>x] = y  # this warns
# frame is unsafe here
```

and

``` python
# WORKS (but using .ix is still better)
temp = frame[columnone]
temp[frame[columntwo]>x] = y  # this warns
# frame is unsafe here, but temp is safe
frame = temp
# now frame is safe
```

no these are treated the same

their is an is_copy flag that is set on the created frames which propogate 

however they can act differently if the reference variable goes out of scope

that said in a single dtype case u can get away with this

but my recommendation still holds to always heed the warning and not use potentially unsafe constructs

Following up on this, I just ran:

``` python
dataframe.loc[dataframe[colname]==colvalue, newcolname]=1
```

and got 

```
C:\Anaconda\lib\site-packages\pandas\core\indexing.py:245: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self.obj[key] = np.nan
```

Isn't this what I'm supposed to do? (irrespective of the warning - it does appear to have done the right thing on my ~50k row dataframe)

(Once I can write an example of how to do this in a way that always works, I will submit a PR as this is super unclear to me and I bet others too)

well, pls `show_versions()`. Its possible its from a prior statement (and just got checked here, if that is the case its a bug). You modifying something that is a view to something else. You will have to show a self-contained example.

Sorry for the lack of info before - assuming my use of `.loc` ought to work, I will see if I can construct an end-to-end example that triggers the warning

```
In [2]: pandas.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.7.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 58 Stepping 9, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.15.1
nose: 1.3.3
Cython: 0.20.1
numpy: 1.9.1
scipy: 0.14.0
statsmodels: 0.5.0
IPython: 2.1.0
sphinx: 1.2.2
patsy: 0.3.0
dateutil: 1.5
pytz: 2014.9
bottleneck: 0.8.0
tables: 3.1.1
numexpr: 2.3.1
matplotlib: 1.3.1
openpyxl: 1.8.5
xlrd: 0.9.3
xlwt: 0.7.5
xlsxwriter: 0.5.5
lxml: 3.3.5
bs4: 4.3.1
html5lib: None
httplib2: None
apiclient: None
rpy2: None
sqlalchemy: 0.9.4
pymysql: 0.6.2.None
psycopg2: None
```

Answer - my code looks like this:

``` python
dataframe=loadDataFrame()
dataframe=dataframe[colnames] # don't need all the columns
... code code code...
dataframe.loc[dataframe[colname]==colvalue, newcolname]=1 # triggers warning
```

Replacing the offending line with

``` python
dataframe=dataframe.drop([col for col in dataframe.columns if col not in colnames], axis=1)
```

means I don't get the warning

So, seems like the documentation 'fix' in this case (or rather helpful pointer) is that the warning can be triggered by an action that takes place some way away from the line that raises the warning - thanks for your help!

well the warning is exactly correct.

you are taking a 'view' of the dataframe (e.g. the `df = df[colnames]`)
This wont' trigger a copy

when you are setting a value via loc you are then effectiviely setting BOTH the original and the new one. This is what the error is guarding against, namely propogating of the view.

simply enough to 

This actually acomplished quite a bit, it provides you with a new smaller object, and the old one gets cleaned up (and not copy warnings),
`df = df[colnames_i_care_about].copy()`

is the proper idiom here

or you don't even need that at all

just directly

`df.loc[row_indexer,column] = value`

this ONLY sets that 1 column so the rest don't matter

Further you can also `usecols=....` when you are reading/loading the frame from disk.

One of my users has code similar to this:

```
df1 = DataFrame({'x': Series(['a','b','c']), 'y': Series(['d','e','f'])})
df2 = df1[['x']]
df2['y'] = ['g', 'h', 'i']
```

And gets the warning. However, in the code base, lines 2 and 3 are quite far removed. So it's almost impossible to realize the warning stems from line 2, (because the warning happens on execution of line 3, far away). I don't know if anything can be done about this, but it can be a head scratcher to track down the source of the warnings.

(Or, even why there's a warning here, without understanding the subtle semantics of copy vs view)


Resolving patch:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -1265,6 +1265,14 @@ def _check_setitem_copy(self, stacklevel=4, t='setting', force=False):
             except:
                 pass
 
+            # we might be a false positive
+            try:
+                if self.is_copy().shape == self.shape:
+                    self.is_copy = None
+                    return
+            except:
+                pass
+
             # a custom message
             if isinstance(self.is_copy, string_types):
                 t = self.is_copy
@@ -1344,8 +1352,9 @@ def take(self, indices, axis=0, convert=True, is_copy=True):
         result = self._constructor(new_data).__finalize__(self)
 
         # maybe set copy if we didn't actually change the index
-        if is_copy and not result._get_axis(axis).equals(self._get_axis(axis)):
-            result._set_is_copy(self)
+        if is_copy:
+            if not result._get_axis(axis).equals(self._get_axis(axis)):
+                result._set_is_copy(self)
 
         return result
 


Test patch:
diff --git a/pandas/tests/test_indexing.py b/pandas/tests/test_indexing.py
--- a/pandas/tests/test_indexing.py
+++ b/pandas/tests/test_indexing.py
@@ -3751,14 +3751,6 @@ def f():
         assert_series_equal(s,df.iloc[:,0].order())
         assert_series_equal(s,df[0].order())
 
-        # operating on a copy
-        df = pd.DataFrame({'a': list(range(4)), 'b': list('ab..'), 'c': ['a', 'b', np.nan, 'd']})
-        mask = pd.isnull(df.c)
-
-        def f():
-            df[['c']][mask] = df[['b']][mask]
-        self.assertRaises(com.SettingWithCopyError, f)
-
         # false positives GH6025
         df = DataFrame ({'column1':['a', 'a', 'a'], 'column2': [4,8,9] })
         str(df)
@@ -3790,6 +3782,24 @@ def f():
             df['C'][2] = 'foo'
         self.assertRaises(com.SettingWithCopyError, f)
 
+    def test_setting_with_copy_bug(self):
+
+        # operating on a copy
+        df = pd.DataFrame({'a': list(range(4)), 'b': list('ab..'), 'c': ['a', 'b', np.nan, 'd']})
+        mask = pd.isnull(df.c)
+
+        def f():
+            df[['c']][mask] = df[['b']][mask]
+        self.assertRaises(com.SettingWithCopyError, f)
+
+        # invalid warning as we are returning a new object
+        # GH 8730
+        df1 = DataFrame({'x': Series(['a','b','c']), 'y': Series(['d','e','f'])})
+        df2 = df1[['x']]
+
+        # this should not raise
+        df2['y'] = ['g', 'h', 'i']
+
     def test_detect_chained_assignment_warnings(self):
 
         # warnings