Repository: pandas-dev/pandas

Issue:
BUG?: .sample() sometimes returning a view not a copy. 
I've realized the `.sample()` function is sometimes returning a view rather than a copy. This will cause the "setting on a view/copy" error:

```
df = pd.DataFrame({'col1':[5,6,7], 'col2':['a','b','c']})
a_sample = df.sample(1)
a_sample['col1'] = 'z'
```

I believe the behavior is the result of it using the `.take()` function, which is what the `.sample()` command uses to pull the sample the passed frame. 

I'd like to suggest either:
- replace `.take()` with `.iloc()`
- change `.take()` to `.take().copy()` if that's faster (@shoyer suggested `.take()` would be a faster -- is `.take().copy()`?)

Thoughts?



Discussion:
both `.iloc` and `.take` _can_ be views on the underlying data, depending on how selections happen and if there are multi-dtypes involved. This is the point of the `SettingWithCopyError`.
Internally, you could always just do:

```
if result.is_view:
    result = result.copy()
return result
```

to guarantee that it will always be a copy

Of course the reason to do this is that people will simply try to mutate this and assume it should work. 

Ok, patch coming!

> both .iloc and .take can be views on the underlying data, depending on how selections happen and if there are multi-dtypes involved.

@jreback This behavior is very surprising to me. I assumed that pandas follows the NumPy rules for copies/views with `iloc`/`take`. That is, `take` always returns a copy and `iloc` returns views only if all indexers are slices or integers.

If we have places where we are trying to convert array indexers into slices so we can do a view in `iloc`/`take`, those should be eliminated.

This may be a naive question, but I've been struggling a little with copy/view issues lately, so seems worth asking: is that kind of data-structure-dependent variation in behavior worth having? For example, it doesn't seem like the following commands should be different, but one (the later) gives the setting with copy error:

```
df = pd.DataFrame({'col1':[2,3,4], 'col2':[5,6,7]})
i = df.loc[2]
i['col1']=3


df = pd.DataFrame({'col1':[2,3,4], 'col2':['a','b','c']})
i = df.loc[2]
i['col1']=3
```

(after discovering this behavior, I've literally spent the last two days going back through a major project i'm working on, setting `pd.set_option('mode.chained_assignment','raise')`, and re-running days worth of code to make sure it's safe).

@jreback @shoyer This worth bringing up in separate issue?

@shoyer not sure what you are talking about. The semantics are the same as numpy.

If you have a single dtype, then take will give you a view if  you have a slice (or sometimes when selecting by integer). Same for iloc. Multi-dtypes will _always_ give you a copy.

This has always been true.

@nickeubank this is the entire point of the `SettingWithCopyError`. You can end up with what you think is a view. But then you decide to add a copy which is a different dtype. Then it becomes a copy.

@shoyer I see what you are talking about now. So `.take` does seem to give a copy always, while `.iloc` will give a view if possible. Very odd, I always though `.take` will give a view.

@jreback I guess my question is more: is this really desirable as a property from a user-design perspective? The `SettingWithCopyError` does indeed flag when this happens, and missing this is my own error. 

But from a general design perspective, is it ideal to expect users to always be mindful of the dtypes of columns they aren't directly manipulating in their `DataFrames` to know what `pandas` is doing / are we comfortable with a non-exception warning being the only safeguard? Surely we've all lost track of warning outputs in a stream of output, no?

(I'm thinking about my colleagues in the social sciences -- this just seems the kind of subtle and potentially unnecessarily complicated behavior that will really throw them for a loop)

@nickeubank this is the fundamental design of numpy. Its less than ideal but generally not a problem. The warning never used to exists and we would get questions, like: why does this not set?

e.g.

`df['foo']['bar'] = 10`.

When it would work if it was a view, but the second someone added another dtype it would then not work.

The thing is you HAVE/WANT to always use views as they are extremely cheap. Otherwise you end up copying everywhere which is bad.


Resolving patch:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -2004,9 +2004,14 @@ def sample(self, n=None, frac=None, replace=False, weights=None, random_state=No
             Sample with or without replacement. Default = False.
         weights : str or ndarray-like, optional
             Default 'None' results in equal probability weighting.
+            If passed a Series, will align with target object on index. Index 
+            values in weights not found in sampled object will be ignored and
+            index values in sampled object not in weights will be assigned 
+            weights of zero. 
             If called on a DataFrame, will accept the name of a column
             when axis = 0.
-            Weights must be same length as axis being sampled.
+            Unless weights are a Series, weights must be same length as axis 
+            being sampled.
             If weights do not sum to 1, they will be normalized to sum to 1.
             Missing values in the weights column will be treated as zero.
             inf and -inf values not allowed.
@@ -2019,7 +2024,7 @@ def sample(self, n=None, frac=None, replace=False, weights=None, random_state=No
 
         Returns
         -------
-        Same type as caller.
+        A new object of same type as caller.
         """
 
         if axis is None:
@@ -2034,6 +2039,10 @@ def sample(self, n=None, frac=None, replace=False, weights=None, random_state=No
         # Check weights for compliance
         if weights is not None:
 
+            # If a series, align with frame
+            if isinstance(weights, pd.Series):                
+                weights = weights.reindex(self.axes[axis])                
+
             # Strings acceptable if a dataframe and axis = 0
             if isinstance(weights, string_types):
                 if isinstance(self, pd.DataFrame):
@@ -2063,7 +2072,10 @@ def sample(self, n=None, frac=None, replace=False, weights=None, random_state=No
 
             # Renormalize if don't sum to 1
             if weights.sum() != 1:
-                weights = weights / weights.sum()
+                if weights.sum() != 0:
+                    weights = weights / weights.sum()
+                else:
+                    raise ValueError("Invalid weights: weights sum to zero")
 
             weights = weights.values
 
@@ -2082,7 +2094,8 @@ def sample(self, n=None, frac=None, replace=False, weights=None, random_state=No
             raise ValueError("A negative number of rows requested. Please provide positive value.")
 
         locs = rs.choice(axis_length, size=n, replace=replace, p=weights)
-        return self.take(locs, axis=axis)
+        return self.take(locs, axis=axis, is_copy=False)
+
 
     _shared_docs['pipe'] = ("""
         Apply func(self, \*args, \*\*kwargs)


Test patch:
diff --git a/pandas/tests/test_generic.py b/pandas/tests/test_generic.py
--- a/pandas/tests/test_generic.py
+++ b/pandas/tests/test_generic.py
@@ -374,7 +374,7 @@ def test_sample(self):
 
             self._compare(o.sample(frac=0.7,random_state=np.random.RandomState(test)),
                           o.sample(frac=0.7, random_state=np.random.RandomState(test)))
-
+ 
 
         # Check for error when random_state argument invalid.
         with tm.assertRaises(ValueError):
@@ -415,6 +415,10 @@ def test_sample(self):
             bad_weights = [0.5]*11
             o.sample(n=3, weights=bad_weights)
 
+        with tm.assertRaises(ValueError):
+            bad_weight_series = Series([0,0,0.2])
+            o.sample(n=4, weights=bad_weight_series)
+            
         # Check won't accept negative weights
         with tm.assertRaises(ValueError):
             bad_weights = [-0.1]*10
@@ -431,6 +435,16 @@ def test_sample(self):
             weights_with_ninf[0] =  -np.inf
             o.sample(n=3, weights=weights_with_ninf)
 
+        # All zeros raises errors
+        zero_weights = [0]*10
+        with tm.assertRaises(ValueError):
+            o.sample(n=3, weights=zero_weights)
+
+        # All missing weights
+        nan_weights = [np.nan]*10
+        with tm.assertRaises(ValueError):
+            o.sample(n=3, weights=nan_weights)
+
 
         # A few dataframe test with degenerate weights.
         easy_weight_list = [0]*10
@@ -496,7 +510,6 @@ def test_sample(self):
         assert_frame_equal(df.sample(n=1, axis='index', weights=weight),
                            df.iloc[5:6])
 
-
         # Check out of range axis values
         with tm.assertRaises(ValueError):
             df.sample(n=1, axis=2)
@@ -527,6 +540,26 @@ def test_sample(self):
         assert_panel_equal(p.sample(n=3, random_state=42), p.sample(n=3, axis=1, random_state=42))
         assert_frame_equal(df.sample(n=3, random_state=42), df.sample(n=3, axis=0, random_state=42))
 
+        # Test that function aligns weights with frame
+        df = DataFrame({'col1':[5,6,7], 'col2':['a','b','c'], }, index = [9,5,3])
+        s = Series([1,0,0], index=[3,5,9])
+        assert_frame_equal(df.loc[[3]], df.sample(1, weights=s))
+
+        # Weights have index values to be dropped because not in 
+        # sampled DataFrame
+        s2 = Series([0.001,0,10000], index=[3,5,10])
+        assert_frame_equal(df.loc[[3]], df.sample(1, weights=s2))
+
+        # Weights have empty values to be filed with zeros
+        s3 = Series([0.01,0], index=[3,5])
+        assert_frame_equal(df.loc[[3]], df.sample(1, weights=s3))
+
+        # No overlap in weight and sampled DataFrame indices
+        s4 = Series([1,0], index=[1,2])
+        with tm.assertRaises(ValueError):
+            df.sample(1, weights=s4)
+
+
     def test_size_compat(self):
         # GH8846
         # size property should be defined