Repository: pandas-dev/pandas

Issue:
Slowness for DataFrame.where/mask when dtype=bool
``` python
data = np.random.randn(10000, 500)
df = pd.DataFrame(data)
df = df.where(df > 0) # create nans
bools = df > 0

reg_mask = bools.mask(pd.isnull(df)) # walltime 10s
float_mask = bools.astype(float).mask(pd.isnull(df)) # walltime 200ms
tm.assert_frame_equal(reg_mask, float_mask) # success
```

I found that while masking boolean DataFrames, converting to float speeds up the process considerably. When you're masking at least one row in each column, the outputs are equivalent. 

Note, the mask is slow even when you're only masking one column

``` python
data = np.random.randn(10000, 500)
df = pd.DataFrame(data)
df.ix[:, 0] = np.nan
bools = df > 0

reg_mask = bools.mask(pd.isnull(df)) # walltime 10s
# dtypes: bool(499), float64(1)
```



Discussion:
This is a _fundamental_ problem in that since we alway try to give you back the same dtype you came in with (if possible), this results in a pretty slow conversion. The masking function e.g. `df>0` is operating on a boolean array here; when I evaluate in numpy it changs it to `float64`, which is fine.

Then we do the downcasting back to bools where possible (which in this particular case is degenerate in that we try to convert it back , but since the `nans` are basically in every column, we can't, so we end up with floats.

This conversion is item-by-item as `a-prioi` don't know which columns can be converted (well I do know, but again have to convert item-by-item); The degenerate case is where 1/2 of the columns can be converted. So can get some gains back.

This will also happen with `integers` , e.g. any dtype that cannot hold `nans`. 

So not really sure what do do here; it makes the code a fair bit more complicated to handle the _splitting_ case better than we do now.

any thoughts?

## Explanation

I'm not sure if you're familiar with the internals at all, but one reason is that the `bool` dtype cannot hold `NaN`s. Under the hood pandas checks to see if the `Block` (a `Block` is a container for a chunk of data of a uniform `dtype`) being operated on can hold `NaN`. If it can't then it will loop over each item (the `Int64Index` elements in this case) and perform the operation on each column.

## Numbers

**the following numbers are from my machine, using** [`lprun`](https://pypi.python.org/pypi/line_profiler/)
- `take` on a single column of `values` (the method calling frame): 4.3 ms
- `take` on a single column of `cond` (the input boolean frame): 9.6 ms
- number of times each of those is called: 500
- total time = `500 times * (4.3 ms + 9.6 ms) == 6950 ms == 6.95 s`

## Conclusion

`take` is 3 orders of magnitude slower (!) than the equivalent slice expression (which times in at about 19 us). it might be hard to get around this when using `MultiIndex`es but maybe a special case for block items that are `Int64index`es can be put in.

@jreback what about building a slice expression rather than using take, e.g.,

``` python
x.take([0], axis=0) == x[[0]]
x.take([0], axis=1) == x[:, [0]]
```

in general

``` python
def slice_it(x, ind, axis):
    slices = [slice(None)] * x.ndim
    slices[axis] = ind
    return x[tuple(slices)]
```

i think should work, but i haven't tried it yet

nothing to do with `take` per se; its all about trying to convert the resulting calculation

take is quite fast, but when called 500 times there is a lot of overhead

I can do it, but as I said, I am not sure the additional code complexity is worth it.

Basically you have to figure out what you can a-priori downcast back to the original type; right now we are doing this item-by-item which is why its slow;

I will give another try to see what I can do

hm then lprun is wrong. 97 percent of the computation is spent on the takes

i get it down to 235ish ms when using a slice a 29x speedup seems worth it no?

ok...pretty much fixed (these are for 1000 cols btw)
Its still somewhat slower because have to process 2 resulting blocks but much better
(was 400+ ms before)

```
In [6]: bools.mask(mask)
Out[6]: 
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1000 entries, 0 to 999
Columns: 500 entries, 0 to 499
dtypes: float64(500)

In [7]: %timeit bools.mask(mask)
100 loops, best of 3: 14.8 ms per loop

In [8]: %timeit bools.astype(float).mask(mask)
100 loops, best of 3: 3.01 ms per loop
```

@cpcloud the issue is take was called once for each item, but calling it once doesn't really help because of all of the other stuff that is going one)

@cpcloud and you get all of the tests to pass?????

no no no that was a quick and dirty crap hack to prove my point :/ i can do the general case and try it out

i will add the slicing, not hard

and actually...the reason the original is so fast is the it is handled by `numexpr`!

the `bool` dtype is not handled by `numexpr`

tests pass. if u want i can push it

(there's an issue with pytables in travis)


Resolving patch:
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -558,42 +558,38 @@ def func(c,v,o):
                     result.fill(np.nan)
                     return result
 
-        def create_block(result, items, transpose=True):
+        # see if we can operate on the entire block, or need item-by-item
+        result = func(cond,values,other)
+        if self._can_hold_na:
+
             if not isinstance(result, np.ndarray):
                 raise TypeError('Could not compare [%s] with block values'
                                 % repr(other))
 
-            if transpose and is_transposed:
+            if is_transposed:
                 result = result.T
 
             # try to cast if requested
             if try_cast:
                 result = self._try_cast_result(result)
 
-            return make_block(result, items, self.ref_items)
+            return make_block(result, self.items, self.ref_items)
 
-        # see if we can operate on the entire block, or need item-by-item
-        if not self._can_hold_na:
-            axis = cond.ndim-1
-            result_blocks = []
-            for item in self.items:
-                loc  = self.items.get_loc(item)
-                item = self.items.take([loc])
-                v    = values.take([loc],axis=axis)
-                c    = cond.take([loc],axis=axis)
-                o    = other.take([loc],axis=axis) if hasattr(other,'shape') else other
-
-                result = func(c,v,o)
-                if len(result) == 1:
-                    result = np.repeat(result,self.shape[1:])
-
-                result = _block_shape(result,ndim=self.ndim,shape=self.shape[1:])
-                result_blocks.append(create_block(result, item, transpose=False))
-
-            return result_blocks
-        else:
-            result = func(cond,values,other)
-            return create_block(result, self.items)
+        # might need to separate out blocks
+        axis = cond.ndim-1
+        cond = cond.swapaxes(axis,0)
+        mask = np.array([ cond[i].all() for i in enumerate(range(cond.shape[0]))],dtype=bool)
+
+        result_blocks = []
+        for m in [mask, ~mask]:
+            if m.any():
+                items = self.items[m]
+                slices = [slice(None)] * cond.ndim
+                slices[axis] = self.items.get_indexer(items)
+                r = self._try_cast_result(result[slices])
+                result_blocks.append(make_block(r.T, items, self.ref_items))
+
+        return result_blocks
 
 class NumericBlock(Block):
     is_numeric = True
@@ -2429,7 +2425,22 @@ def _lcd_dtype(l):
     elif have_bool:
         return np.dtype(bool)
     elif have_int and not have_float and not have_complex:
-        return _lcd_dtype(counts[IntBlock])
+
+        # if we are mixing unsigned and signed, then return
+        # the next biggest int type (if we can)
+        lcd = _lcd_dtype(counts[IntBlock])
+        kinds = set([ i.dtype.kind for i in counts[IntBlock] ])
+        if len(kinds) == 1:
+            return lcd
+
+        if lcd == 'uint64' or lcd == 'int64':
+            return np.dtype('int64')
+
+        # return 1 bigger on the itemsize if unsinged
+        if lcd.kind == 'u':
+            return np.dtype('int%s' % (lcd.itemsize*8*2))
+        return lcd
+    
     elif have_dt64 and not have_float and not have_complex:
         return np.dtype('M8[ns]')
     elif have_complex:
diff --git a/vb_suite/frame_methods.py b/vb_suite/frame_methods.py
--- a/vb_suite/frame_methods.py
+++ b/vb_suite/frame_methods.py
@@ -177,3 +177,18 @@ def f(K=500):
 """
 
 frame_xs_col = Benchmark('df.xs(50000,axis = 1)', setup)
+
+## masking
+setup = common_setup + """
+data = np.random.randn(1000, 500)
+df = DataFrame(data)
+df = df.where(df > 0) # create nans
+bools = df > 0
+mask = isnull(df)
+"""
+
+mask_bools = Benchmark('bools.mask(mask)', setup,
+                         start_date=datetime(2013,1,1))
+
+mask_floats  = Benchmark('bools.astype(float).mask(mask)', setup,
+                         start_date=datetime(2013,1,1))


Test patch:
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -271,6 +271,16 @@ def test_getitem_boolean_casting(self):
         expected = Series({'float64': 6, 'int32' : 1, 'int64' : 1})
         assert_series_equal(result, expected)
 
+        # where dtype conversions
+        # GH 3733
+        df = DataFrame(data = np.random.randn(100, 50))
+        df = df.where(df > 0) # create nans
+        bools = df > 0
+        mask = isnull(df)
+        expected = bools.astype(float).mask(mask)
+        result = bools.mask(mask)
+        assert_frame_equal(result,expected)
+
     def test_getitem_boolean_list(self):
         df = DataFrame(np.arange(12).reshape(3, 4))
 
@@ -7568,8 +7578,10 @@ def test_where(self):
 
         def _safe_add(df):
             # only add to the numeric items
-            return DataFrame(dict([ (c,s+1) if issubclass(s.dtype.type, (np.integer,np.floating)) else (c,s) for c, s in df.iteritems() ]))
-
+            def is_ok(s):
+                return issubclass(s.dtype.type, (np.integer,np.floating)) and s.dtype != 'uint8'
+            return DataFrame(dict([ (c,s+1) if is_ok(s) else (c,s) for c, s in df.iteritems() ]))
+        
         def _check_get(df, cond, check_dtypes = True):
             other1 = _safe_add(df)
             rs = df.where(cond, other1)
@@ -7605,7 +7617,7 @@ def _check_get(df, cond, check_dtypes = True):
         def _check_align(df, cond, other, check_dtypes = True):
             rs = df.where(cond, other)
             for i, k in enumerate(rs.columns):
-                v = rs[k]
+                result = rs[k]
                 d = df[k].values
                 c = cond[k].reindex(df[k].index).fillna(False).values
 
@@ -7613,12 +7625,16 @@ def _check_align(df, cond, other, check_dtypes = True):
                     o = other
                 else:
                     if isinstance(other,np.ndarray):
-                        o = Series(other[:,i],index=v.index).values
+                        o = Series(other[:,i],index=result.index).values
                     else:
                         o = other[k].values
 
                 new_values = d if c.all() else np.where(c, d, o)
-                assert_series_equal(v, Series(new_values,index=v.index))
+                expected = Series(new_values,index=result.index)
+
+                # since we can't always have the correct numpy dtype
+                # as numpy doesn't know how to downcast, don't check
+                assert_series_equal(result, expected, check_dtype=False)
 
             # dtypes
             # can't check dtype when other is an ndarray
@@ -9894,14 +9910,14 @@ def test_as_matrix_lcd(self):
         self.assert_(values.dtype == np.float16)
 
         values = self.mixed_int.as_matrix(['A','B','C','D'])
-        self.assert_(values.dtype == np.uint64)
+        self.assert_(values.dtype == np.int64)
 
         values = self.mixed_int.as_matrix(['A','D'])
         self.assert_(values.dtype == np.int64)
 
         # guess all ints are cast to uints....
         values = self.mixed_int.as_matrix(['A','B','C'])
-        self.assert_(values.dtype == np.uint64)
+        self.assert_(values.dtype == np.int64)
 
         values = self.mixed_int.as_matrix(['A','C'])
         self.assert_(values.dtype == np.int32)