Repository: statsmodels/statsmodels

Issue:
Incorrect implementation of confusion matrix in pred_table for Logit model
The confusion matrix given by the pred_table() function gives the incorrect value. E.g.

```
y = np.array([[1],
       [0],
       [1],
       [0],
       [0],
       [0],
       [0],
       [0],
       [0],
       [0]])
res = Logit(y,np.ones((10,1))).fit()
res.pred_table()
```

Outputs the following:

```
array([[ 0.,  8.],
       [ 0.,  2.]])
```

This output is wrong on multiple accounts: 
- Given that label 1 is considered true and label 0 is considered false:

```
TP = 0, FP = 2, FN = 0, TN = 8
```
- The correct predictions for each class should fall on the diagonal, however, in this case the correct predictions appear to be on the off diagonal. 
- As mentioned in Issue #1577 `tp, fp, fn, tn = map(float, pred_table.flatten())`, if the true class is 0. However, when using Logit model the true class is almost always 1. And even if the true class is 0, then also TP is going to be cm[0,1] and not cm[0,0], where cm is the confusion matrix. 

The model, the actual confusion matrix, considering label 1 as the positive class, should be:

```
[[0,2],
[0,8]]
#representing the traditional
[[TP, FN],
[FP, TN]]
```

Alternatively, it can also be simply:

```
[[8,0],
[2,0]]
#representing the sklearn implementation
[[Actual = 0, Predicted = 0, Actual = 0, Predicted = 1],
[Actual = 1, Predicted = 0, Actual = 1, Predicted = 1]]
# Ensuring that the diagonal is the correct class predictions counts. 
```



Discussion:
@napsternxg  Thanks for reporting

Yes, that's a bug, which AFAICS causes problems when a row or column is zero.
The method just uses numpy histogram2d with bins=2, which produces the wrong bin limits if there are not both zeros and ones

```
>>> np.histogram2d(res.model.endog, pred, bins=2)
(array([[ 0.,  8.],
       [ 0.,  2.]]), array([ 0. ,  0.5,  1. ]), array([-0.5,  0. ,  0.5]))
```

to fix this, I think it should be enough to use explicit bin boundaries

```
>>> b1 = np.array([ 0. ,  0.5,  1. ])
>>> np.histogram2d(res.model.endog, pred, bins=(b1, b1))
(array([[ 8.,  0.],
       [ 2.,  0.]]), array([ 0. ,  0.5,  1. ]), array([ 0. ,  0.5,  1. ]))
```

I think using the threshold in the bin directly would make the 0-1 prediction unnecessary, but I don't remember how numpy histogram handles values exactly at the bounds (in case of ties and points on the threshold).
`b1 = np.array([ 0. ,  threshold,  1. ])`

Based on browsing the unit tests, test coverage for pred_table looks good. So, this problem might be only in this (untested) edge case.

@napsternxg  Do you want to prepare a PR with the fix and your example as unit test?

@josef-pkt 
I can prepare a PR and send by next week. However, I think we can use sklearn's API for creating the confusion matrix which will make it more consistent for users coming form ML background. I think in Issue #1650 there was already discussion to some code from sklearn api for doing this. 

Another option would be explicitly construct the Confusion matrix in the format such that the diagonal elements contain the true predictions for each class. In Logit case it would be just 2 classes  such that 0 is the first class. The code would also be more readable as opposed to using np.histogram2d. 

What do you think ?

the problem is that any refactoring or enhancements will need decisions and review, while fixing the current implementation for the edge case with unit test can be merged right away.

I think histogram usage is quite fine, as long as it has explicit intervals. It's a common usage for counting a binned continuous variable.

Extensions along #1650 would be a good contribution, of course, but I'm not usually using this functionality and I'm not really familiar with what scikit-learn is doing in this area. 
So, I don't have a strong opinion about details and would mainly review for overall consistency with the statsmodels pattern.

Overall predictive evaluation measures is not yet a strong, consistently supported area in statsmodels. I added the `eval_measures` for continuous data, but my usecase was more Monte Carlo studies than predictive cross-validation, plus some helper function for mainly internal usage.

This means, that it would be very useful if you or someone else is pushing into this area. 

MNLogit has the same problem if a zero column is at the boundary. There is a unit test for it with zeros in the middle. Zero columns at the boundary mess up the bin edges for histogram2d.


Resolving patch:
diff --git a/statsmodels/discrete/discrete_model.py b/statsmodels/discrete/discrete_model.py
--- a/statsmodels/discrete/discrete_model.py
+++ b/statsmodels/discrete/discrete_model.py
@@ -2748,7 +2748,9 @@ def pred_table(self, threshold=.5):
         model = self.model
         actual = model.endog
         pred = np.array(self.predict() > threshold, dtype=float)
-        return np.histogram2d(actual, pred, bins=2)[0]
+        bins = np.array([0, 0.5, 1])
+        return np.histogram2d(actual, pred, bins=bins)[0]
+
 
     def summary(self, yname=None, xname=None, title=None, alpha=.05,
                 yname_list=None):
@@ -2956,11 +2958,12 @@ def pred_table(self):
         pred_table[i,j] refers to the number of times "i" was observed and
         the model predicted "j". Correct predictions are along the diagonal.
         """
-        J = self.model.J
+        ju = self.model.J - 1  # highest index
         # these are the actual, predicted indices
-        idx = lzip(self.model.endog, self.predict().argmax(1))
+        #idx = lzip(self.model.endog, self.predict().argmax(1))
+        bins = np.concatenate(([0], np.linspace(0.5, ju - 0.5, ju), [ju]))
         return np.histogram2d(self.model.endog, self.predict().argmax(1),
-                              bins=J)[0]
+                              bins=bins)[0]
 
     @cache_readonly
     def bse(self):


Test patch:
diff --git a/statsmodels/discrete/tests/test_discrete.py b/statsmodels/discrete/tests/test_discrete.py
--- a/statsmodels/discrete/tests/test_discrete.py
+++ b/statsmodels/discrete/tests/test_discrete.py
@@ -1407,6 +1407,22 @@ def test_formula_missing_exposure():
     assert_raises(ValueError, sm.Poisson, df.Foo, df[['constant', 'Bar']],
                   exposure=exposure)
 
+
+def test_binary_pred_table_zeros():
+    # see 2968
+    nobs = 10
+    y = np.zeros(nobs)
+    y[[1,3]] = 1
+
+    res = Logit(y, np.ones(nobs)).fit(disp=0)
+    expected = np.array([[ 8.,  0.], [ 2.,  0.]])
+    assert_equal(res.pred_table(), expected)
+
+    res = MNLogit(y, np.ones(nobs)).fit(disp=0)
+    expected = np.array([[ 8.,  0.], [ 2.,  0.]])
+    assert_equal(res.pred_table(), expected)
+
+
 if __name__ == "__main__":
     import nose
     nose.runmodule(argv=[__file__, '-vvs', '-x', '--pdb'],