Repository: pandas-dev/pandas

Issue:
BUG: partial boolean indexing fails with dtype = int
this fails because we are putmasking on a  dtype=int ndarray
need to cast if this is the case (in series.where)

```
import pandas
df = pandas.DataFrame(index=[1,2])
df['test'] = [1,2]
df['test'][[True, False]] = pandas.Series([0],index=[1])

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-9-7738933c68bb> in <module>()
----> 1 df['test'][[True,False]] = pandas.Series([0],index=[1])

/mnt/home/jreback/pandas/pandas/core/series.pyc in __setitem__(self, key, value)
    677         if _is_bool_indexer(key):
    678             key = self._check_bool_indexer(key)
--> 679             self.where(~key,value,inplace=True)
    680         else:
    681             self._set_with(key, value)

/mnt/home/jreback/pandas/pandas/core/series.pyc in where(self, cond, other, inplace)
    630             raise ValueError('Length of replacements must equal series length')
    631 
--> 632         np.putmask(ser, ~cond, other)
    633 
    634         return None if inplace else ser

TypeError: array cannot be safely cast to required type

```



Discussion:
@stephenwlin we didn't close this for some reason....any thoughts?

ok i can take a look

@jreback I thought about this and there's probably a one or two line fix that can handle this particular case...but I think the most general solution here might be to implement a function called `_maybe_unify` which takes a list (or tuple, maybe...maybe either or) of dtypes and returns the most general dtype that can hold all of them.

This would probably be useful in a lot of other places where similar unifications are required, like joins or sets (I think a lot of this logic is duplicated in different places right now and not complete either, mostly only handling ints and floats cases). For example, I'm fairly sure that right now, `__setitem__` on DataFrame (either directly or through the `ix` property) either doesn't even allow assigning values that don't fit in the existing column types, or downcasts them implicit to the existing types, rather than upcasting appropriately to fit them which is probably the right thing to do instead: implementing such upcasts would be a lot easier with such a utility function.

(Possibly `_maybe_promote` could even be rewritten to be in terms of `_maybe_unify`, since most of what the former is doing is basically a restricted form of the latter...however, there's also some extra work in computing/modifying a `fill_value` so it might take some thought to keep it clean and while preserving that.)

The one thing I can think of that might be kind of tricky would be the case of datetime64[ns]: right now, they don't upcast correctly to objects, so you can't unify them with other types: instead, you want to allow other objects (like dates, string, etc.) to implicit cast to datetime64[ns] on assignment, if possible, as well as convert NaN to NaT. This is currently handled as a special case in `__setitem__` I think...maybe it can be done with a more general solution that doesn't introduce local hacks.

What do you think? Overkill?

(On second thought, might not be worth it to handle the >2 dtypes case...I don't think it happens anywhere practically)

I think this is related:

http://stackoverflow.com/questions/15077885/typeerror-when-changing-an-integer-data-series-in-pandas/15078445#15078445

I tried to fix this, and my solution would work, but have to change the data and dtype of a series inplace is currently not possible, in np 1.7 would work though http://docs.scipy.org/doc/numpy/reference/generated/numpy.copyto.html#numpy.copyto

(once series is refactored to have a _data as its internals, then this is easy)

got it....in place updating

`
ser.dtype = result.dtype
ser[:] = result
`


Resolving patch:
diff --git a/pandas/core/common.py b/pandas/core/common.py
--- a/pandas/core/common.py
+++ b/pandas/core/common.py
@@ -722,6 +722,20 @@ def _maybe_promote(dtype, fill_value=np.nan):
     return dtype, fill_value
 
 
+def _maybe_upcast_putmask(result, mask, other):
+    """ a safe version of put mask that (potentially upcasts the result
+        return the result and a changed flag """
+    try:
+        np.putmask(result, mask, other)
+    except:
+        # our type is wrong here, need to upcast
+        if (-mask).any():
+            result, fill_value = _maybe_upcast(result, copy=True)
+            np.putmask(result, mask, other)
+            return result, True
+
+    return result, False
+
 def _maybe_upcast(values, fill_value=np.nan, copy=False):
     """ provide explicty type promotion and coercion
         if copy == True, then a copy is created even if no upcast is required """
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -201,7 +201,7 @@ def na_op(x, y):
                 mask = notnull(xrav)
                 result[mask] = op(xrav[mask], y)
 
-            np.putmask(result, -mask, NA)
+            result, changed = com._maybe_upcast_putmask(result,-mask,np.nan)
             result = result.reshape(x.shape)
 
         return result
diff --git a/pandas/core/series.py b/pandas/core/series.py
--- a/pandas/core/series.py
+++ b/pandas/core/series.py
@@ -70,7 +70,8 @@ def na_op(x, y):
             else:
                 mask = notnull(x)
                 result[mask] = op(x[mask], y)
-            np.putmask(result, -mask, pa.NA)
+
+            result, changed = com._maybe_upcast_putmask(result,-mask,pa.NA)
 
         return result
 
@@ -680,7 +681,13 @@ def where(self, cond, other=nan, inplace=False):
         if len(other) != len(ser):
             raise ValueError('Length of replacements must equal series length')
 
-        np.putmask(ser, ~cond, other)
+        result, changed = com._maybe_upcast_putmask(ser,~cond,other)
+        if changed:
+
+            # need to actually change ser here
+            if inplace:
+                ser.dtype = result.dtype
+                ser[:] = result
 
         return None if inplace else ser
 


Test patch:
diff --git a/pandas/tests/test_series.py b/pandas/tests/test_series.py
--- a/pandas/tests/test_series.py
+++ b/pandas/tests/test_series.py
@@ -2943,6 +2943,17 @@ def test_asof_more(self):
         result = s.asof(s.index[0])
         self.assertEqual(result, s[0])
 
+    def test_cast_on_putmask(self):
+
+        # GH 2746
+
+        # need to upcast
+        s = Series([1,2],index=[1,2],dtype='int64')
+        s[[True, False]] = Series([0],index=[1],dtype='int64')
+        expected = Series([0,2],index=[1,2],dtype='float64')
+
+        assert_series_equal(s, expected)
+
     def test_astype_cast_nan_int(self):
         df = Series([1.0, 2.0, 3.0, np.nan])
         self.assertRaises(ValueError, df.astype, np.int64)