Repository: scipy/scipy

Issue:
The sparse .sum() method with uint8 dtype does not act like the corresponding numpy method.
With NumPy calling the `.sum()` method along an axis upcasts the dtype from uint8 to uint64, probably to prevent rolling over back to 0:

```
In [9]: np.matrix([[1, 255],
                   [0, 0]], dtype=np.uint8).sum(1)
Out[9]: matrix([256,   0], dtype=uint64)
```

With the sparse package in SciPy, there is no upcasting, so 255 + 1 rolls over to 0. For the same data we get:

```
 In [11]: sp.csc_matrix(np.array([[1, 255],
                                  [0, 0]], dtype=np.uint8)).sum(1)
 Out[11]: 
 matrix([[0],
         [0]], dtype=uint8)
```

The shapes are different too...

So should Sparse copy numpy's upcasting behavior? 



Discussion:
Am I not able to add labels to issues I created? This issue should be labeled with scipy.sparse

Yes, I think this should be changed to match Numpy behavior. Numpy also IIRC used to work like this, but the accumulator type was bumped up to a larger one at some point. The expected behavior probably is: use platform integer (or, the spmatrix type if it is larger) as the accumulator.

Also, the reduction methods should gain a dtype argument.

I think there we need mixed input and output type versions of the reduction routines in sparsetools. This again grows the SWIG wrappers...

Deprecation of the old behavior is probably not doable here. Someone relying on the accumulator being of a smaller type is probably not so common occurrence (in contrast, there have been a number of people complaining on stackoverflow etc being surprised about the wraparound), so I think it is possible to change this behavior.

So... The `.mean()` and `.sum()` (and probably other) methods in sparse return results that are the same dtype as the spmatrix. In numpy these methods upcast to the platform default float or integer.

@pv What do you mean by "reduction methods"?

I don't think I understand you correctly, you say deprecation is "not doable", but then "it is possible to change this behavior."

"Reduction" is Numpy jargon for sum(), prod(), ...

By "not doable" I mean that it is not possible to make this change in stages, as the change of the accumulator type will cause backward incompatibility in any case, and trying to lessen this impact makes things worse.

Okay, I am including a change in the accumulator type in my next PR for bool dtype support.

@cowlicks @pv It seems that the `.sum()` method of SciPy sparse matrices is still not in line with the NumPy  `.sum()` method in terms of upcasting behavior. The original example of this issue still results in the same discrepancy (integer overflow for sparse matrices, upcasting for dense matrices) (SciPy 0.18.0.dev0+b2eea7c):

```
In [19]: np.matrix([[1, 255],
                    [0, 0]], dtype=np.uint8).sum(1)
Out[19]: 
matrix([[256],
        [  0]], dtype=uint64)


In [20]: sp.csc_matrix(np.array([[1, 255],
                                 [0, 0]], dtype=np.uint8)).sum(1)
Out[20]: 
matrix([[0],
        [0]], dtype=uint8)

```

Fixable via `np.int_ -> np.integral`, `np.float_ -> np.floating` in the lines touched by gh-2549

If I remember it correctly, it should be `np.integer` right ? Please correct me if I am wrong.

@maniteja123  `np.integer` seems to be the correct fix, yes. 

I can easily submit a PR with the fix, but I am unsure what test to add to sparse/tests/test_base.py. The `.math_dtypes` attributes of the matrix class based tests only consist of `[np.bool_, np.int_, np.float_, np.complex_]`, so `.test_sum()` never tests for integer upcasting behavior.

@Tiepies looking at my previous PR I think `test_base.py` is okay.

I have not seen `np.integer` and `np.floating` before, so maybe they are new. Which would explain why they are not used in the sparse tests.

@cowlicks your PR indeed tested for this, but PR #5213 reduced the type coverage by replacing `self.checked_dtypes` with `self.math_dtypes`, which means the upcasting behavior is not tested for anymore. I am not comfortable with touching the tests, because I do not fully grasp the reason for this change.

Yep, that PR was my doing. The whole dtype testing situation is still a bit of a mess, unfortunately, but [this comment](https://github.com/scipy/scipy/pull/5213#issuecomment-138990519) should hopefully clear up some confusion.

The takeaway is that we haven't ever been testing for correct upcasting behavior, though we thought that we were before my PR.

Possible fixes include:
1. adding `np.int8` to `math_dtypes` (will likely require fixing other bugs to make things pass)
2. changing `test_sum` to use `checked_dtypes` instead of `math_dtypes`

In the long run, we want to have option (1), but for the scope of this PR, option (2) should be sufficient.


Resolving patch:
diff --git a/scipy/sparse/base.py b/scipy/sparse/base.py
--- a/scipy/sparse/base.py
+++ b/scipy/sparse/base.py
@@ -705,11 +705,10 @@ def sum(self, axis=None):
         # Mimic numpy's casting.
         if np.issubdtype(self.dtype, np.float_):
             res_dtype = np.float_
-        elif (np.issubdtype(self.dtype, np.int_) or
-              np.issubdtype(self.dtype, np.bool_)):
-                res_dtype = np.int_
-        elif np.issubdtype(self.dtype, np.complex_):
-            res_dtype = np.complex_
+        elif self.dtype.kind in 'ib':
+            res_dtype = np.int_
+        elif self.dtype.kind == 'u':
+            res_dtype = np.uint
         else:
             res_dtype = self.dtype
 
@@ -737,13 +736,13 @@ def mean(self, axis=None):
                 np.issubdtype(self.dtype, np.integer) or
                 np.issubdtype(self.dtype, np.bool_)):
             res_dtype = np.float_
-        elif np.issubdtype(self.dtype, np.complex_):
-            res_dtype = np.complex_
         else:
             res_dtype = self.dtype
 
         if axis is None:
-            return self.sum(None) * 1.0 / (self.shape[0]*self.shape[1])
+            m, n = self.shape
+            return self.astype(res_dtype).sum() / np.matrix(m * n,
+                                                            dtype=res_dtype)
 
         if axis < 0:
             axis += 2
diff --git a/scipy/sparse/compressed.py b/scipy/sparse/compressed.py
--- a/scipy/sparse/compressed.py
+++ b/scipy/sparse/compressed.py
@@ -564,10 +564,17 @@ def sum(self, axis=None):
         elif (not hasattr(self, 'blocksize') and
               axis in self._swap(((1, -1), (0, 2)))[0]):
             # faster than multiplication for large minor axis in CSC/CSR
-            dtype = self.dtype
-            if np.issubdtype(dtype, np.bool_):
-                dtype = np.int_
-            ret = np.zeros(len(self.indptr) - 1, dtype=dtype)
+            # Mimic numpy's casting.
+            if np.issubdtype(self.dtype, np.float_):
+                res_dtype = np.float_
+            elif self.dtype.kind in 'ib':
+                res_dtype = np.int_
+            elif self.dtype.kind == 'u':
+                res_dtype = np.uint
+            else:
+                res_dtype = self.dtype
+            ret = np.zeros(len(self.indptr) - 1, dtype=res_dtype)
+
             major_index, value = self._minor_reduce(np.add)
             ret[major_index] = value
             ret = np.asmatrix(ret)


Test patch:
diff --git a/scipy/sparse/tests/test_base.py b/scipy/sparse/tests/test_base.py
--- a/scipy/sparse/tests/test_base.py
+++ b/scipy/sparse/tests/test_base.py
@@ -749,16 +749,15 @@ def test_sum(self):
         dat_1 = np.matrix([[0, 1, 2],
                            [3, -4, 5],
                            [-6, 7, 9]])
-        dat_2 = np.random.rand(40, 40)
+        dat_2 = np.random.rand(5, 5)
         dat_3 = np.array([[]])
         dat_4 = np.zeros((40, 40))
-        dat_5 = sparse.rand(40, 40, density=1e-2).A
+        dat_5 = sparse.rand(5, 5, density=1e-2).A
         matrices = [dat_1, dat_2, dat_3, dat_4, dat_5]
 
         def check(dtype, j):
             dat = np.matrix(matrices[j], dtype=dtype)
             datsp = self.spmatrix(dat, dtype=dtype)
-
             assert_array_almost_equal(dat.sum(), datsp.sum())
             assert_equal(dat.sum().dtype, datsp.sum().dtype)
             assert_array_almost_equal(dat.sum(axis=None), datsp.sum(axis=None))
@@ -772,7 +771,7 @@ def check(dtype, j):
             assert_array_almost_equal(dat.sum(axis=-1), datsp.sum(axis=-1))
             assert_equal(dat.sum(axis=-1).dtype, datsp.sum(axis=-1).dtype)
 
-        for dtype in self.math_dtypes:
+        for dtype in self.checked_dtypes:
             for j in range(len(matrices)):
                 yield check, dtype, j
 
@@ -796,7 +795,7 @@ def check(dtype):
             assert_array_almost_equal(dat.mean(axis=-1), datsp.mean(axis=-1))
             assert_equal(dat.mean(axis=-1).dtype, datsp.mean(axis=-1).dtype)
 
-        for dtype in self.math_dtypes:
+        for dtype in self.checked_dtypes:
             yield check, dtype
 
     def test_expm(self):
@@ -3972,7 +3971,15 @@ def spmatrix(self, D, **kwargs):
         NC = construct(arg1, **kwargs)
 
         # check that result is valid
-        assert_allclose(NC.A, M.A)
+        if NC.dtype in [np.float32, np.complex64]:
+            # For single-precision floats, the differences between M and NC
+            # that are introduced by the extra operations involved in the
+            # construction of NC necessitate a more lenient tolerance level
+            # than the default.
+            rtol = 1e-05
+        else:
+            rtol = 1e-07
+        assert_allclose(NC.A, M.A, rtol=rtol)
 
         # check that at least one explicit zero
         if has_zeros: