Repository: sunpy/sunpy

Issue:
Exporting SDO FITS Files without Image Data Arrays
<!--
We know asking good questions takes effort, and we appreciate your time.
Thank you.

Please be aware that everyone has to follow our code of conduct:
https://github.com/sunpy/sunpy/blob/master/CODE_OF_CONDUCT.rst

Also that these comments are hidden when you submit this github issue.

Please have a search on our GitHub repository to see if a similar issue has already been posted.
If a similar issue is closed, have a quick look to see if you are satisfied by the resolution.
If not please go ahead and open an issue!
-->

### Description
<!-- Provide a general description of the issue. -->
I'd like to propose a change to the JSOC export system to help users handle image files that are requested but missing and cannot be downloaded. In addition to affecting the JSOC export web-page users, this change would affect users of software that uses the JSOC export-system CGI. This includes various existing scripts that use URL-opening libraries or linux commands to access these CGI scripts. As such, this will affect users of the drms/securedrms Python package and, consequently, SunPy users that download SDO data with fido or drms directly. 

The Problem
-----------
Internally, the JSOC stores image metadata (i.e., FITS keyword data) separately from the actual image data, and there are cases where metadata exist, like the time of an observation, but no image exists. An example is the JSOC data storage that results when an eclipse prevents SDO from capturing an image of the sun. The observation time and other metadata are stored at the JSOC, but no image exists so no image is stored. Storing metadata separate from image data can lead to an export-system user having access to the metadata, but not having access to an image. Software that queries the JSOC database to obtain metadata often assumes that because metadata exist, so do the corresponding images. The software does know ahead of time if images are missing, so this is a reasonable supposition. The software will then attempt to download the non-existent images and fail. Things get worse if the software has requested many images, but only one image is non-existent - the entire export request will fail.

The JSOC does have a not-well-known means for checking on the presence of an image file, but download-software generally has not been implemented to use it. And even if software were to use this method, it is a bit inefficient and cumbersome to have to make two queries every time you want to download a range of image data that is mostly present. Why not ask us for a range of image data and we give you what we have within that range, telling you when an expected image is missing?

Blah, Blah, Blah (skippable details)
------------------------------------
DRMS provides an INCONSISTENT means for notifying download-software of ONE cause of this problem. For certain series a metadata keyword named QUALITY exists. The value of this keyword is a 32-bit bitmap, with one bit indicating whether an image exists or not. I say QUALITY is inconsistent because some DRMS data series have the QUALITY keyword, and some do not. And the meaning of the various bits is series-specific. I will set up a different thread to deal with the inconsistency of QUALITY. It is an important problem that must also be dealt with. But for the present discussion, let's assume your software has attempted to download a 'missing' image.

There are other reasons why an image might be missing. The most likely one is an image file is on tape, but not on disk - offline data. The JSOC has a finite amount of disk space so not all data can be online at the same time. Exporting offline data is currently not supported due to the potential for the need for manual intervention to place data online. In addition, there are other less likely reasons to be in this situation: a disk crashes, a network switch dies, a file system goes belly up. The point is, no matter how careful one is, there is no way to prevent with 100% certainty an export user from finding themselves in a missing-image scenario.

Proposal
--------
Here is what I'd like to do: if an export user has access to image metadata, but not the actual image, I'd like to send the user a file that has the image metadata, but no image. In this way, as long as the normal HTTP mechanisms work, the requestor will not be in the situation where they attempted to download an image file, which failed, potentially causing the request for many images to fail also. At the very least, the user will end up with a file that contains the metadata. At the moment, we are focusing on data series with a FITS-file container, especially since this scenario, where metadata exist but no image exists, is going to almost always concern FITS files. So in the FITS case, the user will end up with a FITS file that has FITS keywords (but NAXIS will be 0 since no image exists), but no image array. However, the principle should apply to all types of data-series file containers. For a user of drms.py/securedrms.py, the client.export() call will ultimately result in the download of FITS keywords, just as the client.query() call does. 

If I were to make this change, would this help the SunPy community? What problems might it cause?




Discussion:
Thanks for the detailed post @davisite I finally got some time to experiment with the ideas.

Firstly, I think this is a great idea, and I don't forsee any issues with our drms / JSOCClient / Fido code stack because of these proposed changes.

My concerns are mostly linked to the `map` submodule, which reads the FITS files. I wanted to know how we would handle a user giving `sunpy.map.Map` a list of FITS files (the results of a export etc) where one of many aren't valid maps (no image data = not valid as a map).

To think through this I did some experimenting, firstly I tried to generate a file with `NAXIS = 0`, this seems impossible with astropy, so I went into the FITS standard, and it does indeed specify that NAXIS can only be in the range 1-999. What astropy does when writing a file with no data in the HDU is write `NAXIS = 1, NAXIS` = 0`, so assuming your code generated data like this I think we would be fine.

When reading these files, they were silently ignored by the map loader, which means that there isn't an issue for users passing in a list of files like you are proposing, where some might be missing data so that's good :+1:. However, I consider that a bug so I will be opening a PR soon to probably change the default behaviour to that raising an error, but there is an easy flag to suppress the error and open all files which are good.

So TL;DR I think this plan is great, and there should be no issues on the sunpy side with you implementing it!

Thanks a lot for coming to us and getting our feedback!

oh one other question, would you be RICE compressing these files as well, for consistency? I am not sure what a length 0 RICE compressed array looks like :thinking: 
Hey @Cadair 

Hopefully my comment wasn't too long (LOL!), and hopefully it is helpful.

Actually, I believe the FITS standard does allow for NAXIS to be 0. From the Standard:
**
NAXIS keyword.
The value field shall contain a non-negative integer no greater than 999 representing the number of axes in the associated data array.  A value of zero signifies that no data follow the header in the HDU.
**
If there is a data array, then NAXIS can't be 0. If you use fits_write_imghdr() in CFITSIO, you can set naxis = 0 in the naxis argument. I make NAXIS = 0 fits files with DRMS. I'm not sure what happens if you try to make a NAXIS = 0 FITS file with fits_write_img() though - I'm guessing it won't work, and maybe that is what astropy uses.

cheers

👍 sounds like all is settled here - I'm going to tag https://github.com/sunpy/sunpy/pull/4426 as closing this issue when it's merged, so we properly handle NAXIS=0,1 cases.

Resolving patch:
diff --git a/sunpy/map/map_factory.py b/sunpy/map/map_factory.py
--- a/sunpy/map/map_factory.py
+++ b/sunpy/map/map_factory.py
@@ -24,6 +24,7 @@
     NoMatchError,
     ValidationFunctionError,
 )
+from sunpy.util.exceptions import NoMapsInFileError, SunpyUserWarning
 from sunpy.util.functools import seconddispatch
 from sunpy.util.metadata import MetaDict
 from sunpy.util.types import DatabaseEntryType
@@ -172,6 +173,10 @@ def _read_file(self, fname, **kwargs):
                 data = filedata
                 meta = MetaDict(filemeta)
                 new_pairs.append((data, meta))
+
+        if not new_pairs:
+            raise NoMapsInFileError(f"Found no HDUs with >= 2D data in '{fname}'.")
+
         return new_pairs
 
     def _validate_meta(self, meta):
@@ -185,7 +190,7 @@ def _validate_meta(self, meta):
         else:
             return False
 
-    def _parse_args(self, *args, **kwargs):
+    def _parse_args(self, *args, silence_errors=False, **kwargs):
         """
         Parses an args list into data-header pairs.
 
@@ -236,7 +241,12 @@ def _parse_args(self, *args, **kwargs):
         # Note that this list can also contain GenericMaps if they are directly given to the factory
         data_header_pairs = []
         for arg in args:
-            data_header_pairs += self._parse_arg(arg, **kwargs)
+            try:
+                data_header_pairs += self._parse_arg(arg, **kwargs)
+            except NoMapsInFileError as e:
+                if not silence_errors:
+                    raise
+                warnings.warn(f"One of the arguments failed to parse with error: {e}", SunpyUserWarning)
 
         return data_header_pairs
 
@@ -320,7 +330,7 @@ def __call__(self, *args, composite=False, sequence=False, silence_errors=False,
         Extra keyword arguments are passed through to `sunpy.io.read_file` such
         as `memmap` for FITS files.
         """
-        data_header_pairs = self._parse_args(*args, **kwargs)
+        data_header_pairs = self._parse_args(*args, silence_errors=silence_errors, **kwargs)
         new_maps = list()
 
         # Loop over each registered type and check to see if WidgetType
@@ -339,7 +349,7 @@ def __call__(self, *args, composite=False, sequence=False, silence_errors=False,
                     ValidationFunctionError, MapMetaValidationError) as e:
                 if not silence_errors:
                     raise
-                warnings.warn(f"One of the data, header pairs failed to validate with: {e}")
+                warnings.warn(f"One of the data, header pairs failed to validate with: {e}", SunpyUserWarning)
 
         if not len(new_maps):
             raise RuntimeError('No maps loaded')
diff --git a/sunpy/util/exceptions.py b/sunpy/util/exceptions.py
--- a/sunpy/util/exceptions.py
+++ b/sunpy/util/exceptions.py
@@ -7,10 +7,17 @@
 
 from astropy.utils.exceptions import AstropyWarning
 
-__all__ = ["SunpyWarning", "SunpyUserWarning", "SunpyDeprecationWarning",
+__all__ = ["NoMapsInFileError",
+           "SunpyWarning", "SunpyUserWarning", "SunpyDeprecationWarning",
            "SunpyPendingDeprecationWarning", "SunpyMetadataWarning"]
 
 
+class NoMapsInFileError(Exception):
+    """
+    An error raised when a file is opened and no maps are found.
+    """
+
+
 class SunpyWarning(AstropyWarning):
     """
     The base warning class from which all Sunpy warnings should inherit.


Test patch:
diff --git a/sunpy/map/tests/test_map_factory.py b/sunpy/map/tests/test_map_factory.py
--- a/sunpy/map/tests/test_map_factory.py
+++ b/sunpy/map/tests/test_map_factory.py
@@ -16,7 +16,7 @@
 import sunpy
 import sunpy.data.test
 import sunpy.map
-from sunpy.util.exceptions import SunpyUserWarning
+from sunpy.util.exceptions import NoMapsInFileError, SunpyUserWarning
 
 filepath = pathlib.Path(sunpy.data.test.rootdir)
 a_list_of_many = [os.fspath(f) for f in pathlib.Path(filepath, "EIT").glob("*")]
@@ -267,3 +267,16 @@ def test_map_list_urls_cache():
 def test_sources(file, mapcls):
     m = sunpy.map.Map(file)
     assert isinstance(m, mapcls)
+
+
+def test_no_2d_hdus(tmpdir):
+    # Create a fake FITS file with a valid header but 1D data
+    tmp_fpath = str(tmpdir / 'data.fits')
+    with fits.open(AIA_171_IMAGE, ignore_blank=True) as hdul:
+        fits.writeto(tmp_fpath, np.arange(100), hdul[0].header)
+
+    with pytest.raises(NoMapsInFileError, match='Found no HDUs with >= 2D data'):
+        sunpy.map.Map(tmp_fpath)
+
+    with pytest.warns(SunpyUserWarning, match='One of the arguments failed to parse'):
+        sunpy.map.Map([tmp_fpath, AIA_171_IMAGE], silence_errors=True)