Repository: statsmodels/statsmodels

Issue:
logit marginal effects
The dummy-option for the compution of the marginal effects from a logisitc regression result does not seem to work properly. When setting the option to True the marginal effect of the continues variable changes, not that of the dummy variable.


    from __future__ import division
    import numpy as np
    from statsmodels.discrete.discrete_model import Logit
 
    # generate data (the last column is a dummy variable)
    n_rows = 10e5
    x = np.c_[np.ones(n_rows),np.random.uniform(-20,20,size=(n_rows,)), np.random.chisquare(15,size=(n_rows,)), np.random.binomial(1, 0.7, n_rows)]
    b_true = np.array([1.2, -0.0143, 0.096, 0.65])

    xb = np.sum(x*b_true, axis=1)
    p = 1/(1 + np.exp(-xb))
    u = np.random.rand(n_rows)
    y = np.zeros(n_rows)
    y[u<p] = 1

    # estimate
    logit = Logit(y, x)
    res = logit.fit(method='bfgs')
    # marginal effects at the mean
    me = res.get_margeff(at='mean').summary()
    # computing them "by hand"
    mean = np.mean(x,axis=0)
    p_hat = 1/(1+np.exp(-np.sum(mean*res.params)))
    me_own = p_hat*(1-p_hat)*params[1:]
    # the marginal effects agree

    # Using the dummy option the me of x_3 should change to:
    p_1 = 1/(1+ np.exp(-np.sum(np.append(mean[:-1],1)*res.params)))
    p_0 = 1/(1+ np.exp(-np.sum(np.append(mean[:-1],0)*res.params)))    
    me_dummy = p_1 - p_0

    # however, using the dummy-option changes the marginal effect of x_2 (not of x_3)
    me_sm_dummy =  res.get_margeff(at='mean',dummy = True).summary()

statsmodels version: 0.6.1
python 2.7



Discussion:
To make it more precise.  
The following assert should hold, according to this

```
me_sm_dummy =  res.get_margeff(at='mean',dummy = True)
assert_allclose(me_sm_dummy.margeff[-1], me_dummy, rtol=1e-13)
``` 

looks like an indexing bug, or a bug when trying to identify binary columns.  
But I'm not familiar enough with the implementation details to make a reliable guess.
and the others should stay unchanged according to the docstring
`assert_allclose(me_sm_dummy.margeff[:-1], me_sm_dummy.margeff[:-1], rtol=1e-13)`

(aside `rtol=1e-13` is my usual starting point if precision depends on floating point operations with not very large rounding errors.)
just a thought @jktis  Can you try your example with the constant as last parameter instead of first?

An obvious candidate for a bug could be mixing up exog indices with and without constant. Stata uses prepend=False, and maybe that's the only cases that are in the unit tests.
indeed. without the constant, the dummy=True option adjusts the correct parameter and computes it correctly.
 a bit more in this direction const_idx is correctly zero, in the example

```
def _get_dummy_index(X, const_idx):
    ...
    # adjust back for a constant because effects doesn't have one
    if const_idx is not None:
        dummy_ind[dummy_ind > const_idx] -= 1
```

but I didn't figure out yet if the main computation uses original exog, or exog without constant column.

I see three test cases and all have prepend=False.
@jktis thanks for reporting and checking.

This looks like a serious bug, especially given that we switched to prepend=True by default (in contrast to Stata)
When I comment out the if with `dummy_ind[dummy_ind > const_idx] -= 1` then all unit test still pass, and a unit test adjusted with prepend=True also passes.  
This would indicate that the `_get_dummy_index` is only used in the case with all columns, or that there is insufficient test coverage to see where it breaks.

`_get_count_index` has the same adjustment to the index, and so will most likely have the same bug.

Resolving patch:
diff --git a/statsmodels/discrete/discrete_margins.py b/statsmodels/discrete/discrete_margins.py
--- a/statsmodels/discrete/discrete_margins.py
+++ b/statsmodels/discrete/discrete_margins.py
@@ -70,9 +70,7 @@ def _isdummy(X):
 def _get_dummy_index(X, const_idx):
     dummy_ind = _isdummy(X)
     dummy = True
-    # adjust back for a constant because effects doesn't have one
-    if const_idx is not None:
-        dummy_ind[dummy_ind > const_idx] -= 1
+
     if dummy_ind.size == 0: # don't waste your time
         dummy = False
         dummy_ind = None # this gets passed to stand err func
@@ -107,9 +105,7 @@ def _iscount(X):
 def _get_count_index(X, const_idx):
     count_ind = _iscount(X)
     count = True
-    # adjust back for a constant because effects doesn't have one
-    if const_idx is not None:
-        count_ind[count_ind > const_idx] -= 1
+
     if count_ind.size == 0: # don't waste your time
         count = False
         count_ind = None # for stand err func


Test patch:
diff --git a/statsmodels/discrete/tests/test_discrete.py b/statsmodels/discrete/tests/test_discrete.py
--- a/statsmodels/discrete/tests/test_discrete.py
+++ b/statsmodels/discrete/tests/test_discrete.py
@@ -889,6 +889,55 @@ def test_dummy_exog2(self):
         assert_almost_equal(me.margeff_se,
                 self.res2.margeff_dummy_atexog2_se, DECIMAL_4)
 
+
+class TestLogitNewtonPrepend(CheckMargEff):
+    # same as previous version but adjusted vor add_constant prepend=True
+    # bug #3695
+    @classmethod
+    def setup_class(cls):
+        data = sm.datasets.spector.load()
+        data.exog = sm.add_constant(data.exog, prepend=True)
+        cls.res1 = Logit(data.endog, data.exog).fit(method="newton", disp=0)
+        res2 = Spector()
+        res2.logit()
+        cls.res2 = res2
+        cls.slice = np.roll(np.arange(len(cls.res1.params)), 1) #.astype(int)
+
+    def test_resid_pearson(self):
+        assert_almost_equal(self.res1.resid_pearson,
+                            self.res2.resid_pearson, 5)
+
+    def test_nodummy_exog1(self):
+        me = self.res1.get_margeff(atexog={1 : 2.0, 3 : 1.})
+        assert_almost_equal(me.margeff,
+                self.res2.margeff_nodummy_atexog1, DECIMAL_4)
+        assert_almost_equal(me.margeff_se,
+                self.res2.margeff_nodummy_atexog1_se, DECIMAL_4)
+
+    def test_nodummy_exog2(self):
+        me = self.res1.get_margeff(atexog={2 : 21., 3 : 0}, at='mean')
+        assert_almost_equal(me.margeff,
+                self.res2.margeff_nodummy_atexog2, DECIMAL_4)
+        assert_almost_equal(me.margeff_se,
+                self.res2.margeff_nodummy_atexog2_se, DECIMAL_4)
+
+    def test_dummy_exog1(self):
+        me = self.res1.get_margeff(atexog={1 : 2.0, 3 : 1.}, dummy=True)
+        assert_almost_equal(me.margeff,
+                self.res2.margeff_dummy_atexog1, DECIMAL_4)
+        assert_almost_equal(me.margeff_se,
+                self.res2.margeff_dummy_atexog1_se, DECIMAL_4)
+
+    def test_dummy_exog2(self):
+        me = self.res1.get_margeff(atexog={2 : 21., 3 : 0}, at='mean',
+                dummy=True)
+        assert_almost_equal(me.margeff,
+                self.res2.margeff_dummy_atexog2, DECIMAL_4)
+        assert_almost_equal(me.margeff_se,
+                self.res2.margeff_dummy_atexog2_se, DECIMAL_4)
+
+
+
 class TestLogitBFGS(CheckBinaryResults, CheckMargEff):
     @classmethod
     def setupClass(cls):