Repository: scipy/scipy

Issue:
Test failures for sparse sum tests on 32-bit Python
Getting a large amount of test failures like the one below, I assume that this is due to gh-5907:

```
======================================================================
FAIL: test_base.TestCSC.test_sum(<type 'numpy.uint64'>, 1)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/rgommers/Code/scipy/scipy/sparse/tests/test_base.py", line 767, in check
    assert_equal(dat.sum(axis=0).dtype, datsp.sum(axis=0).dtype)
  File "/home/rgommers/.local/lib/python2.7/site-packages/numpy/testing/utils.py", line 375, in assert_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
 ACTUAL: dtype('uint64')
 DESIRED: dtype('uint32')
```

And also a large set of overflow warnings, all on the same line:

```
/home/rgommers/Code/scipy/scipy/sparse/dok.py:370: RuntimeWarning: overflow encountered in ulonglong_scalars
  result[i] += v * other[j]
```



Discussion:
The test failures are easy to fix by adding `np.can_cast` conditions in the type casting part of the `spmatrix.sum` and `compressed.sum` method definitions. I will submit a pull request with that fix.

The overflow warnings are thrown because #5907 (re-)introduced tests for a wider range of dtypes in `test_base._TestCommon.test_sum`, including unsigned integers. The output of `np.matrix.sum` is the same as that of `dok_matrix.sum`, but only the latter protests about overflow:

```
>>> import numpy as np
>>> from scipy.sparse import dok_matrix
>>> dat_1 = np.matrix([[0, 1, 2],
                       [3, -4, 5],
                       [-6, 7, 9]])
>>> dat = np.matrix(dat_1, dtype = np.uint32)
>>> datsp = dok_matrix(dat, dtype=np.uint32)
>>> dat.sum()
17
>>> datsp.sum()
/home/bram/packages/scipy/scipy/sparse/dok.py:370: RuntimeWarning: overflow encountered in uint_scalars
  result[i] += v * other[j]
17
```

How to appropriately deal with those warnings, I don't know.

One workaround is to use a `numpy.errstate` context manager in the `test_sum` method to ignore the warnings, but I don't know if that is desirable. The fact that `numpy.matrix.sum` does not warn for overflow in these cases is likely related to numpy/numpy#593.


Resolving patch:
diff --git a/scipy/sparse/base.py b/scipy/sparse/base.py
--- a/scipy/sparse/base.py
+++ b/scipy/sparse/base.py
@@ -757,10 +757,11 @@ def sum(self, axis=None):
         # Mimic numpy's casting.
         if np.issubdtype(self.dtype, np.float_):
             res_dtype = np.float_
-        elif self.dtype.kind in 'ib':
-            res_dtype = np.int_
-        elif self.dtype.kind == 'u':
+        elif (self.dtype.kind == 'u' and
+              np.can_cast(self.dtype, np.uint)):
             res_dtype = np.uint
+        elif np.can_cast(self.dtype, np.int_):
+            res_dtype = np.int_
         else:
             res_dtype = self.dtype
 
diff --git a/scipy/sparse/compressed.py b/scipy/sparse/compressed.py
--- a/scipy/sparse/compressed.py
+++ b/scipy/sparse/compressed.py
@@ -567,10 +567,11 @@ def sum(self, axis=None):
             # Mimic numpy's casting.
             if np.issubdtype(self.dtype, np.float_):
                 res_dtype = np.float_
-            elif self.dtype.kind in 'ib':
-                res_dtype = np.int_
-            elif self.dtype.kind == 'u':
+            elif (self.dtype.kind == 'u' and
+                  np.can_cast(self.dtype, np.uint)):
                 res_dtype = np.uint
+            elif np.can_cast(self.dtype, np.int_):
+                res_dtype = np.int_
             else:
                 res_dtype = self.dtype
             ret = np.zeros(len(self.indptr) - 1, dtype=res_dtype)


Test patch:
diff --git a/scipy/sparse/tests/test_base.py b/scipy/sparse/tests/test_base.py
--- a/scipy/sparse/tests/test_base.py
+++ b/scipy/sparse/tests/test_base.py
@@ -758,19 +758,22 @@ def test_sum(self):
         def check(dtype, j):
             dat = np.matrix(matrices[j], dtype=dtype)
             datsp = self.spmatrix(dat, dtype=dtype)
-            assert_array_almost_equal(dat.sum(), datsp.sum())
-            assert_equal(dat.sum().dtype, datsp.sum().dtype)
-            assert_(np.isscalar(datsp.sum(axis=None)))
-            assert_array_almost_equal(dat.sum(axis=None), datsp.sum(axis=None))
-            assert_equal(dat.sum(axis=None).dtype, datsp.sum(axis=None).dtype)
-            assert_array_almost_equal(dat.sum(axis=0), datsp.sum(axis=0))
-            assert_equal(dat.sum(axis=0).dtype, datsp.sum(axis=0).dtype)
-            assert_array_almost_equal(dat.sum(axis=1), datsp.sum(axis=1))
-            assert_equal(dat.sum(axis=1).dtype, datsp.sum(axis=1).dtype)
-            assert_array_almost_equal(dat.sum(axis=-2), datsp.sum(axis=-2))
-            assert_equal(dat.sum(axis=-2).dtype, datsp.sum(axis=-2).dtype)
-            assert_array_almost_equal(dat.sum(axis=-1), datsp.sum(axis=-1))
-            assert_equal(dat.sum(axis=-1).dtype, datsp.sum(axis=-1).dtype)
+            with np.errstate(over='ignore'):
+                assert_array_almost_equal(dat.sum(), datsp.sum())
+                assert_equal(dat.sum().dtype, datsp.sum().dtype)
+                assert_(np.isscalar(datsp.sum(axis=None)))
+                assert_array_almost_equal(dat.sum(axis=None),
+                                          datsp.sum(axis=None))
+                assert_equal(dat.sum(axis=None).dtype,
+                             datsp.sum(axis=None).dtype)
+                assert_array_almost_equal(dat.sum(axis=0), datsp.sum(axis=0))
+                assert_equal(dat.sum(axis=0).dtype, datsp.sum(axis=0).dtype)
+                assert_array_almost_equal(dat.sum(axis=1), datsp.sum(axis=1))
+                assert_equal(dat.sum(axis=1).dtype, datsp.sum(axis=1).dtype)
+                assert_array_almost_equal(dat.sum(axis=-2), datsp.sum(axis=-2))
+                assert_equal(dat.sum(axis=-2).dtype, datsp.sum(axis=-2).dtype)
+                assert_array_almost_equal(dat.sum(axis=-1), datsp.sum(axis=-1))
+                assert_equal(dat.sum(axis=-1).dtype, datsp.sum(axis=-1).dtype)
 
         for dtype in self.checked_dtypes:
             for j in range(len(matrices)):