Repository: statsmodels/statsmodels

Issue:
discrete errors due to incorrect in-place operation
Getting a number of errors like this (with numpy master):

```
======================================================================
ERROR: statsmodels.discrete.tests.test_discrete.TestLogitNewton.test_dummy_dydxmean
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/nose-1.1.2-py2.6.egg/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/Users/rgommers/Code/statsmodels/statsmodels/discrete/tests/test_discrete.py", line 183, in test_dummy_dydxmean
    assert_almost_equal(self.res1.margeff(at='mean', dummy=True),
  File "/Users/rgommers/Code/statsmodels/statsmodels/discrete/discrete_model.py", line 1664, in margeff
    dummy_ind[const_idx:] -= 1
TypeError: Cannot cast ufunc subtract output from dtype('int32') to dtype('bool') with casting rule 'same_kind'
```

Can't see so quickly what `dummy_ind[const_idx:] -= 1` is trying to achieve, but it looks incorrect. `dummy_ind` is a bool array, subtracting 1 from the items in it that are False is odd.



Discussion:
701a44b6e7ea8ee64b5f2e163f8360997bc25397

my impression is that this should be dummy_ind[const_idx] = False
I don't understand why there is a ":" in the slice.

I'm also used to treating boolean as 0,1 integers, so I don't know if numpy master rules out some other usual "calculations" with booleans.

It depends on how you use them:

```
In [5]: x
Out[5]: array([False,  True,  True], dtype=bool)

In [6]: x + 1   # this works fine
Out[6]: array([1, 2, 2])

In [7]: x += 1
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/Users/rgommers/<ipython-input-7-9762fa126317> in <module>()
----> 1 x += 1

TypeError: Cannot cast ufunc add output from dtype('int32') to dtype('bool') with casting rule 'same_kind'
```

The rules have been tightened to catch things that are often bugs. In this case, an integer has to be down cast (down casted?) to a boolean, which in general loses info. So likely a bug. If the code is correct after all, just use `-= False` instead of `-= 1`. But also that is an odd construction.....

I'm going to lump this in with #377.

Ok, looking at this now. dummy_ind was a boolean array, but shouldn't be anymore. If it is, this is a bug. It looks like I forgot to change _isdummy to return np.where(ind)[0] like _iscount, but this makes the tests fail instead of error out. We subtract 1 because the constant is removed (if present) from the marginal effects and therefore the variable indices so every index after the column with the constant gets a -= 1, hence the slice.

Ok, I wasn't thinking. Shouldn't need to slice here like this. I was mixing up the boolean vs. integer indexing.


Resolving patch:
diff --git a/statsmodels/discrete/discrete_model.py b/statsmodels/discrete/discrete_model.py
--- a/statsmodels/discrete/discrete_model.py
+++ b/statsmodels/discrete/discrete_model.py
@@ -82,7 +82,7 @@ def _isdummy(X):
     ind = min & max & remainder
     if X.ndim == 1:
         ind = np.asarray([ind])
-    return ind
+    return np.where(ind)[0]
 
 def _iscount(X):
     """
@@ -149,18 +149,17 @@ def _get_count_effects(effects, exog, count_ind, method, model, params):
 
 
 def _get_dummy_effects(effects, exog, dummy_ind, method, model, params):
-    for i, tf in enumerate(dummy_ind):
-        if tf == True:
-            exog0 = exog.copy() # only copy once, can we avoid a copy?
-            exog0[:,i] = 0
-            effect0 = model.predict(params, exog0)
-            #fittedvalues0 = np.dot(exog0,params)
-            exog0[:,i] = 1
-            effect1 = model.predict(params, exog0)
-            if 'ey' in method:
-                effect0 = np.log(effect0)
-                effect1 = np.log(effect1)
-            effects[i] = (effect1 - effect0).mean() # mean for overall
+    for i in dummy_ind:
+        exog0 = exog.copy() # only copy once, can we avoid a copy?
+        exog0[:,i] = 0
+        effect0 = model.predict(params, exog0)
+        #fittedvalues0 = np.dot(exog0,params)
+        exog0[:,i] = 1
+        effect1 = model.predict(params, exog0)
+        if 'ey' in method:
+            effect0 = np.log(effect0)
+            effect1 = np.log(effect1)
+        effects[i] = (effect1 - effect0).mean() # mean for overall
     return effects
 
 def _effects_at(effects, at, ind):


Test patch: