Repository: gammapy/gammapy

Issue:
Fix test_wcs_nd_map_data_transpose_issue
I think here we forgot to pass `sparse=True` in `test_wcs_nd_map_data_transpose_issue`?

https://github.com/gammapy/gammapy/pull/1346/files#diff-930fa0f94b49b5d6537396fad7c6512aR137-R140

If I change to `sparse=True` in the write call, I get this fail:
```
E           Arrays are not equal
E           
E           x and y nan location mismatch:
E            x: array([[0., 1., 2.],
E                  [0., 0., 0.]], dtype=float32)
E            y: array([[  0.,   1.,   2.],
E                  [ nan,  inf, -inf]])
```

@adonath - Is there a bug?

Should we spend 1-2 days on maps basics and sort out dtype and missing value issues (#1440, #2374, maybe this issue), e.g. with pair-coding & debugging? Or defer all or some to v1.1?

This is a reminder issue, noticed this accidentally while working on
https://github.com/gammapy/gammapy/pull/2350#issuecomment-542754564


Discussion:
I think we should definitely spend a few days (1-2 days might not not sufficient...) on improving `gammapy.maps` before v1.0. However I would prefer to not spend any time on something related to sparse maps. So far I have never worked on sparse maps and I'm not aware of anybody using it. So I think for v1.0 we should just remove sparse maps from the documentation... 
> I would prefer to not spend any time on something related to sparse maps. So far I have never worked on sparse maps and I'm not aware of anybody using it. 

Well, no-one's using it because it's not working yet. But yes, moving this issue to v1.1 milestone.

> So I think for v1.0 we should just remove sparse maps from the documentation...

Maybe. The problem is that the whole gammapy.maps design, with the class hierarchy, and also partly how we use maps in data reduction, only makes sense if there are these extra map types. Otherwise things could be simpler and faster. So I'm leaning a bit to leave as-is and say it's foreseen and to be implemented later. But yes, we should discuss whether to remove for now.

Resolving patch:
diff --git a/gammapy/maps/wcsmap.py b/gammapy/maps/wcsmap.py
--- a/gammapy/maps/wcsmap.py
+++ b/gammapy/maps/wcsmap.py
@@ -270,18 +270,16 @@ def _make_hdu_sparse(data, npix, hdu, header):
 
         if len(shape) == 2:
             data_flat = np.ravel(data)
-            data_flat[~np.isfinite(data_flat)] = 0
-            nonzero = np.where(data_flat > 0)
-            value = data_flat[nonzero].astype(float)
+            non_zero = np.where(~(data_flat == 0))
+            value = data_flat[non_zero].astype(float)
             cols = [
-                fits.Column("PIX", "J", array=nonzero[0]),
+                fits.Column("PIX", "J", array=non_zero[0]),
                 fits.Column("VALUE", "E", array=value),
             ]
         elif npix[0].size == 1:
             shape_flat = shape[:-2] + (shape[-1] * shape[-2],)
             data_flat = np.ravel(data).reshape(shape_flat)
-            data_flat[~np.isfinite(data_flat)] = 0
-            nonzero = np.where(data_flat > 0)
+            nonzero = np.where(~(data_flat == 0))
             channel = np.ravel_multi_index(nonzero[:-1], shape[:-2])
             value = data_flat[nonzero].astype(float)
             cols = [
@@ -295,8 +293,7 @@ def _make_hdu_sparse(data, npix, hdu, header):
             pix = []
             for i, _ in np.ndenumerate(npix[0]):
                 data_i = np.ravel(data[i[::-1]])
-                data_i[~np.isfinite(data_i)] = 0
-                pix_i = np.where(data_i > 0)
+                pix_i = np.where(~(data_i == 0))
                 data_i = data_i[pix_i]
                 data_flat += [data_i]
                 pix += pix_i


Test patch:
diff --git a/gammapy/maps/tests/test_wcsnd.py b/gammapy/maps/tests/test_wcsnd.py
--- a/gammapy/maps/tests/test_wcsnd.py
+++ b/gammapy/maps/tests/test_wcsnd.py
@@ -149,7 +149,7 @@ def test_wcs_nd_map_data_transpose_issue(tmp_path):
     assert_equal(m2.data, data)
 
     # Data should be unmodified after write / read to sparse image format
-    m.write(tmp_path / "sparse.fits.gz")
+    m.write(tmp_path / "sparse.fits.gz", sparse=True)
     m2 = Map.read(tmp_path / "sparse.fits.gz")
     assert_equal(m2.data, data)