Repository: gammapy/gammapy

Issue:
Change SNRCat to gammapy-data
`test_load_catalog_snrcat` is currently failing because it fetches from http://www.physics.umanitoba.ca/snr/SNRcat/ which is currently down.

https://github.com/gammapy/gammapy/blob/6c1c8f48dde5cc47394e21b2985b9f297689f973/gammapy/catalog/tests/test_snrcat.py#L6-L7

There's a TODO in `SNRCat` already to consider putting a copy in `gammapy-extra`:

https://github.com/gammapy/gammapy/blob/6c1c8f48dde5cc47394e21b2985b9f297689f973/gammapy/catalog/snrcat.py#L43

For the second "live" catalog we have that can change any time, gamma-cat`, we currently have a copy in gammapy-extra and access it via gammapy-data:

https://github.com/gammapy/gammapy/blob/6c1c8f48dde5cc47394e21b2985b9f297689f973/gammapy/catalog/gammacat.py#L456

My suggestion would be to also put a fixed stable version of SNRCat in gammapy-extra, to include that in gammapy-data, and to access that from the test.

We could also add a `SNRCat.from_url` to allow both load from gammapy-data or from URL, and in that case would have to decide whether to leave that method untested, or whether to mark the test as "allowed to fail" to not disturb devs and CI when the web service goes down.

@Bultako @adonath - Thoughts?


Discussion:
> My suggestion would be to also put a fixed stable version of SNRCat in gammapy-extra, to include that in gammapy-data, and to access that from the test.

+1
I guess this catalog will be accessed not only from tests, but also in `SourceCatalogSNRcat()`. According to the code in `catalog/gammacat.py` we would need to different csv files:
 - http://www.physics.umanitoba.ca/snr/SNRcat/SNRdownload.php?table=OBS
 - http://www.physics.umanitoba.ca/snr/SNRcat/SNRdownload.php?table=SNR

Would you prefer to have this catalog versioned?
If you all agree and provide me these two files I could do the modif.

@Bultako - I'd like to do this myself, and for v0.13, because changing datasets while @adonath is preparing and doing the release this week is distracting.

There is already the old https://github.com/gammapy/gammapy-extra/blob/master/datasets/catalogs/snrcat.fits.gz and I'd put a new version there instead.

This is mostly a reminder issue for myself - but I also wanted your feedback on the API and testing to put for these two complex cases (gamma-cat and snrcat) of "live and changing" datasets we have. My suggestion is to dump a stable version in gammapy-extra, expose that as gammapy-data and use that for docs and tests. And to add a second method that lets users fetch the latest version from the web, but to not use that in tests or docs, or if we add a test, to mark it as "allowed to fail" because Gammapy CI shouldn't turn red just because http://www.physics.umanitoba.ca/snr/SNRcat/  is down or they change the format. Of course that comes at the price of us probably not noticing when that happens, but IMO that's OK, it's questionable overall if such code should be in Gammapy at all. A more extreme course of action would be to remove it - let me know if you think that's the right thing to do here.
Here's an example showing that currently our CI fails because of `test_load_catalog_snrcat`:

https://dev.azure.com/gammapy/gammapy/_build/results?buildId=903&view=logs&jobId=54b4a527-ead6-5d78-3f02-5346815e0ca2&taskId=86513c84-57b9-57f7-f171-7d82dc57b7aa&lineStart=152&lineEnd=153&colStart=1&colEnd=1
I would definitely add this catalog to `$GAMMAPY_DATA` so that users do not have to worry about broken external data services, so not only for docs and tests. Note that we can expose different versions of a catalog for each release of Gammapy, since we have different [JSON files](https://github.com/gammapy/gammapy-webpage/tree/gh-pages/download/data) (where the lists of datasets are declared), labeled with numbered releases.

If you want to add an additional method to download the catalog from an external url, it's perfectly fine, but I would set it to fetch it from `$GAMMAPY_DATA` by default.






Resolving patch:
diff --git a/gammapy/catalog/__init__.py b/gammapy/catalog/__init__.py
--- a/gammapy/catalog/__init__.py
+++ b/gammapy/catalog/__init__.py
@@ -6,4 +6,3 @@
 from .hawc import *
 from .hess import *
 from .registry import *
-from .snrcat import *
diff --git a/gammapy/catalog/registry.py b/gammapy/catalog/registry.py
--- a/gammapy/catalog/registry.py
+++ b/gammapy/catalog/registry.py
@@ -8,7 +8,6 @@
 from .gammacat import SourceCatalogGammaCat
 from .hawc import SourceCatalog2HWC
 from .hess import SourceCatalogHGPS
-from .snrcat import SourceCatalogSNRcat
 
 __all__ = ["SOURCE_CATALOGS"]
 
@@ -20,6 +19,5 @@
     "4fgl": SourceCatalog4FGL,
     "2fhl": SourceCatalog2FHL,
     "3fhl": SourceCatalog3FHL,
-    "snrcat": SourceCatalogSNRcat,
 }
 """Registry of source catalogs in Gammapy."""
diff --git a/gammapy/catalog/snrcat.py b/gammapy/catalog/snrcat.py
deleted file mode 100644
--- a/gammapy/catalog/snrcat.py
+++ /dev/null
@@ -1,189 +0,0 @@
-# Licensed under a 3-clause BSD style license - see LICENSE.rst
-import numpy as np
-from astropy.coordinates import Angle, SkyCoord
-from astropy.table import Column, Table
-from astropy.utils.data import download_file
-from .core import SourceCatalog, SourceCatalogObject
-
-__all__ = ["SourceCatalogSNRcat", "SourceCatalogObjectSNRcat"]
-
-
-class SourceCatalogObjectSNRcat(SourceCatalogObject):
-    """One source from the SNRcat catalog.
-
-    Catalog is represented by `~gammapy.catalog.SourceCatalogSNRcat`.
-    """
-
-    # TODO: add methods to get position and shape (like for the other cats)
-
-
-class SourceCatalogSNRcat(SourceCatalog):
-    """SNRcat supernova remnant catalog.
-
-    One source is represented by `SourceCatalogObjectSNRcat`.
-
-    `SNRcat <http://www.physics.umanitoba.ca/snr/SNRcat/>`__
-    is a census of high-energy observations of Galactic supernova remnants.
-
-    This function downloads the following CSV-format tables
-    and adds some useful columns and unit information:
-
-    * http://www.physics.umanitoba.ca/snr/SNRcat/SNRdownload.php?table=SNR
-    * http://www.physics.umanitoba.ca/snr/SNRcat/SNRdownload.php?table=OBS
-
-    This only represents a subset of the information available in SNRcat,
-    to get at the rest we would have to scrape their web pages.
-
-    * ``table`` (`~astropy.table.Table`) -- SNR info table
-    * ``obs_table`` (`~astropy.table.Table`) -- High-energy observation info table
-
-    Each table has a ``version`` string containing the download date in the ``table.meta`` dictionary.
-    """
-
-    name = "snrcat"
-    description = "SNRcat supernova remnant catalog."
-    source_object_class = SourceCatalogObjectSNRcat
-
-    def __init__(self, cache=False):
-        # TODO: load from gammapy-extra?
-        # At least optionally?
-        self.snr_table = _fetch_catalog_snrcat_snr_table(cache=cache)
-        self.obs_table = _fetch_catalog_snrcat_obs_table(cache=cache)
-
-        super().__init__(table=self.snr_table)
-
-
-def _fetch_catalog_snrcat_snr_table(cache):
-    url = "http://www.physics.umanitoba.ca/snr/SNRcat/SNRdownload.php?table=SNR"
-    filename = download_file(url, cache=cache)
-
-    # Note: currently the first line contains this comment, which we skip via `header_start=1`
-    table = Table.read(filename, format="ascii.csv", header_start=1, delimiter=";")
-    table.meta["url"] = url
-    table.meta["version"] = _snrcat_parse_download_date(filename)
-
-    # TODO: doesn't work properly ... don't call for now.
-    # _snrcat_fix_na(table)
-
-    table.rename_column("G", "Source_Name")
-
-    table.rename_column("J2000_ra (hh:mm:ss)", "RAJ2000_str")
-    table.rename_column("J2000_dec (dd:mm:ss)", "DEJ2000_str")
-
-    data = Angle(table["RAJ2000_str"], unit="hour").deg
-    index = table.index_column("RAJ2000_str") + 1
-    table.add_column(Column(data=data, name="RAJ2000", unit="deg"), index=index)
-
-    data = Angle(table["DEJ2000_str"], unit="deg").deg
-    index = table.index_column("DEJ2000_str") + 1
-    table.add_column(Column(data=data, name="DEJ2000", unit="deg"), index=index)
-
-    radec = SkyCoord(table["RAJ2000"], table["DEJ2000"], unit="deg")
-    galactic = radec.galactic
-    table.add_column(Column(data=galactic.l.deg, name="GLON", unit="deg"))
-    table.add_column(Column(data=galactic.b.deg, name="GLAT", unit="deg"))
-
-    table.rename_column("age_min (yr)", "age_min")
-    table["age_min"].unit = "year"
-    table.rename_column("age_max (yr)", "age_max")
-    table["age_max"].unit = "year"
-    distance = np.mean([table["age_min"], table["age_max"]], axis=0)
-    index = table.index_column("age_max") + 1
-    table.add_column(Column(distance, name="age", unit="year"), index=index)
-
-    table.rename_column("distance_min (kpc)", "distance_min")
-    table["distance_min"].unit = "kpc"
-    table.rename_column("distance_max (kpc)", "distance_max")
-    table["distance_max"].unit = "kpc"
-    distance = np.mean([table["distance_min"], table["distance_max"]], axis=0)
-    index = table.index_column("distance_max") + 1
-    table.add_column(Column(distance, name="distance", unit="kpc"), index=index)
-
-    table.rename_column("size_radio", "diameter_radio_str")
-    diameter_radio_mean = _snrcat_parse_diameter(table["diameter_radio_str"])
-    index = table.index_column("diameter_radio_str") + 1
-    table.add_column(
-        Column(diameter_radio_mean, name="diameter_radio_mean", unit="arcmin"),
-        index=index,
-    )
-
-    table.rename_column("size_X", "diameter_xray_str")
-    diameter_xray_mean = _snrcat_parse_diameter(table["diameter_xray_str"])
-    index = table.index_column("diameter_xray_str") + 1
-    table.add_column(
-        Column(diameter_xray_mean, name="diameter_xray_mean", unit="arcmin"),
-        index=index,
-    )
-
-    table.rename_column("size_coarse (arcmin)", "diameter_mean")
-    table["diameter_mean"].unit = "arcmin"
-
-    table.rename_column("size_imprecise", "diameter_mean_is_imprecise")
-
-    return table
-
-
-def _fetch_catalog_snrcat_obs_table(cache):
-    url = "http://www.physics.umanitoba.ca/snr/SNRcat/SNRdownload.php?table=OBS"
-    filename = download_file(url, cache=cache)
-
-    # Note: currently the first line contains this comment, which we skip via `header_start=1`
-    table = Table.read(filename, format="ascii.csv", header_start=1, delimiter=";")
-    table.meta["url"] = url
-    table.meta["version"] = _snrcat_parse_download_date(filename)
-
-    # TODO: doesn't work properly ... don't call for now.
-    # _snrcat_fix_na(table)
-
-    return table
-
-
-def _snrcat_fix_na(table):
-    """Fix N/A entries in string columns in SNRcat."""
-    for colname in table.colnames:
-        if isinstance(table[colname][0], str):
-            mask1 = table[colname] == "N / A"
-            mask2 = table[colname] == "N/A"
-            table[colname].mask = mask1 | mask2
-            table[colname].fill_value = ""
-
-
-def _snrcat_parse_download_date(filename):
-    text = open(filename).readline()
-    # Format: "This file was downloaded on 2015-06-07T03:39:53 CDT ..."
-    tokens = text.split()
-    date = tokens[5]
-    return date[:10]
-
-
-def _snrcat_parse_diameter(text_col):
-    """Parse SNRcat diameter string column"""
-    d_means = []
-    for text in text_col:
-        try:
-            # Parse this text field:
-            if "x" in text:
-                a, b = text.split("x")
-                d_major = Angle(a).arcmin
-                d_minor = Angle(b).arcmin
-            else:
-                d_major = Angle(text).arcmin
-                d_minor = d_major
-            d_mean = _snr_mean_diameter(d_major, d_minor)
-        except:
-            # print('Parsing error:', text)
-            d_mean = np.nan
-
-        d_means.append(d_mean)
-
-    return d_means
-
-
-def _snr_mean_diameter(d_major, d_minor):
-    """Compute geometric mean diameter (preserves area)"""
-    diameter = np.sqrt(d_major * d_minor)
-    # If no `d_minor` is given, use `d_major` as mean radius
-    with np.errstate(invalid="ignore"):
-        diameter = np.where(d_minor > 0, diameter, d_major)
-
-    return diameter


Test patch:
diff --git a/gammapy/catalog/tests/test_snrcat.py b/gammapy/catalog/tests/test_snrcat.py
deleted file mode 100644
--- a/gammapy/catalog/tests/test_snrcat.py
+++ /dev/null
@@ -1,25 +0,0 @@
-# Licensed under a 3-clause BSD style license - see LICENSE.rst
-import pytest
-from gammapy.catalog import SourceCatalogSNRcat
-
-
-@pytest.mark.skip(reason="see https://github.com/gammapy/gammapy/issues/2162")
-@pytest.mark.remote_data
-def test_load_catalog_snrcat(tmp_path):
-    snrcat = SourceCatalogSNRcat()
-
-    # Check SNR table
-    table = snrcat.table
-    assert len(table) > 300
-    expected_colnames = ["Source_Name", "RAJ2000"]
-    assert set(expected_colnames).issubset(table.colnames)
-    # Check if catalog can be serialized to FITS
-    table.write(tmp_path / "snrcat_test.fits")
-
-    # Check OBS table
-    table = snrcat.obs_table
-    assert len(table) > 1000
-    expected_colnames = ["SNR_id", "source_id"]
-    assert set(expected_colnames).issubset(table.colnames)
-    # Check if catalog can be serialized to FITS
-    table.write(tmp_path / "obs_test.fits")