Repository: spacetelescope/jwql

Issue:
Dark Monitor not handling absent data
The Dark Monitor is crashing int he case where data is not able to be retrieved:

```
08/24/2020 03:00:56 AM INFO: Working on aperture MIRIM_FULL in miri
08/24/2020 03:00:56 AM INFO: 	No query history for MIRIM_FULL. Beginning search date will be set to 57357.0.
08/24/2020 03:00:56 AM INFO: 	Query times: 57357.0 59085.29225825882
08/24/2020 03:00:57 AM INFO: 	Aperture: MIRIM_FULL, new entries: 12
08/24/2020 03:00:57 AM INFO: 	Sufficient new dark files found for miri, MIRIM_FULL to run the dark monitor.
08/24/2020 03:00:57 AM WARNING: 		Unable to locate jw00619044001_02101_00001_mirimage_dark.fits in filesystem. Not including in processing.
08/24/2020 03:00:57 AM WARNING: 		Unable to locate jw00873010001_02101_00001_mirimage_dark.fits in filesystem. Not including in processing.
08/24/2020 03:00:57 AM WARNING: 		Unable to locate jw00873014001_02101_00001_mirimage_dark.fits in filesystem. Not including in processing.
08/24/2020 03:00:57 AM WARNING: 		Unable to locate jw00873006001_02101_00001_mirimage_dark.fits in filesystem. Not including in processing.
08/24/2020 03:00:57 AM WARNING: 		Unable to locate jw00811002001_02102_00001_mirimage_dark.fits in filesystem. Not including in processing.
08/24/2020 03:00:57 AM WARNING: 		Unable to locate jw00823003001_02102_00001_mirimage_dark.fits in filesystem. Not including in processing.
08/24/2020 03:00:57 AM WARNING: 		Unable to locate jw00619051001_02101_00001_mirimage_dark.fits in filesystem. Not including in processing.
08/24/2020 03:00:57 AM WARNING: 		Unable to locate jw00619058001_02101_00001_mirimage_dark.fits in filesystem. Not including in processing.
08/24/2020 03:00:57 AM WARNING: 		Unable to locate jw00630009001_02101_00001_mirimage_dark.fits in filesystem. Not including in processing.
08/24/2020 03:00:57 AM WARNING: 		Unable to locate jw00619079001_02101_00001_mirimage_dark.fits in filesystem. Not including in processing.
08/24/2020 03:00:57 AM WARNING: 		Unable to locate jw00619086001_02101_00001_mirimage_dark.fits in filesystem. Not including in processing.
08/24/2020 03:00:57 AM WARNING: 		Unable to locate jw00619093001_02101_00001_mirimage_dark.fits in filesystem. Not including in processing.
08/24/2020 03:00:57 AM INFO: 	New_filenames: []
08/24/2020 03:00:57 AM INFO: 	Data dir: /grp/jwst/ins/jwql/outputs/dark_monitor/data/miri_mirim_full
08/24/2020 03:00:57 AM INFO: 	Copied to working dir: []
08/24/2020 03:00:57 AM INFO: 	Not copied: []
08/24/2020 03:00:57 AM CRITICAL: Traceback (most recent call last):
  File "/home/jwqladm/repositories/jwql/jwql/utils/logging_functions.py", line 284, in wrapped
    func(*args, **kwargs)
  File "/home/jwqladm/repositories/jwql/jwql/utils/logging_functions.py", line 249, in wrapped
    func(*args, **kwargs)
  File "/home/jwqladm/repositories/jwql/jwql/instrument_monitors/common_monitors/dark_monitor.py", line 791, in run
    self.process(dark_files)
  File "/home/jwqladm/repositories/jwql/jwql/instrument_monitors/common_monitors/dark_monitor.py", line 517, in process
    self.get_metadata(file_list[0])
IndexError: list index out of range

08/24/2020 03:00:57 AM CRITICAL: CRASHED
```

We should figure out why data is not able to be found, but also make sure that the code can move on to other instruments/apertures in this situation without crashing completely. 


Discussion:
This is a roadblock for #600 

Resolving patch:
diff --git a/jwql/instrument_monitors/common_monitors/dark_monitor.py b/jwql/instrument_monitors/common_monitors/dark_monitor.py
--- a/jwql/instrument_monitors/common_monitors/dark_monitor.py
+++ b/jwql/instrument_monitors/common_monitors/dark_monitor.py
@@ -750,7 +750,7 @@ def run(self):
                 # Check to see if there are enough new files to meet the
                 # monitor's signal-to-noise requirements
                 if len(new_entries) >= file_count_threshold:
-                    logging.info('\tSufficient new dark files found for {}, {} to run the dark monitor.'
+                    logging.info('\tMAST query has returned sufficient new dark files for {}, {} to run the dark monitor.'
                                  .format(self.instrument, self.aperture))
 
                     # Get full paths to the files
@@ -762,29 +762,42 @@ def run(self):
                             logging.warning('\t\tUnable to locate {} in filesystem. Not including in processing.'
                                             .format(file_entry['filename']))
 
-                    # Set up directories for the copied data
-                    ensure_dir_exists(os.path.join(self.output_dir, 'data'))
-                    self.data_dir = os.path.join(self.output_dir,
-                                                 'data/{}_{}'.format(self.instrument.lower(),
-                                                                     self.aperture.lower()))
-                    ensure_dir_exists(self.data_dir)
-
-                    # Copy files from filesystem
-                    dark_files, not_copied = copy_files(new_filenames, self.data_dir)
-
-                    logging.info('\tNew_filenames: {}'.format(new_filenames))
-                    logging.info('\tData dir: {}'.format(self.data_dir))
-                    logging.info('\tCopied to working dir: {}'.format(dark_files))
-                    logging.info('\tNot copied: {}'.format(not_copied))
-
-                    # Run the dark monitor
-                    self.process(dark_files)
-                    monitor_run = True
+                    # If it turns out that the monitor doesn't find enough
+                    # of the files returned by the MAST query to meet the threshold,
+                    # then the monitor will not be run
+                    if len(new_filenames) < file_count_threshold:
+                        logging.info(("\tFilesystem search for the files identified by MAST has returned {} files. "
+                                      "This is less than the required minimum number of files ({}) necessary to run "
+                                      "the monitor. Quitting.").format(len(new_filenames), file_count_threshold))
+                        monitor_run = False
+                    else:
+                        logging.info(("\tFilesystem search for the files identified by MAST has returned {} files.")
+                                     .format(len(new_filenames)))
+                        monitor_run = True
+
+                    if monitor_run:
+                        # Set up directories for the copied data
+                        ensure_dir_exists(os.path.join(self.output_dir, 'data'))
+                        self.data_dir = os.path.join(self.output_dir,
+                                                     'data/{}_{}'.format(self.instrument.lower(),
+                                                                         self.aperture.lower()))
+                        ensure_dir_exists(self.data_dir)
+
+                        # Copy files from filesystem
+                        dark_files, not_copied = copy_files(new_filenames, self.data_dir)
+
+                        logging.info('\tNew_filenames: {}'.format(new_filenames))
+                        logging.info('\tData dir: {}'.format(self.data_dir))
+                        logging.info('\tCopied to working dir: {}'.format(dark_files))
+                        logging.info('\tNot copied: {}'.format(not_copied))
+
+                        # Run the dark monitor
+                        self.process(dark_files)
 
                 else:
-                    logging.info(('\tDark monitor skipped. {} new dark files for {}, {}. {} new files are '
-                                  'required to run dark current monitor.').format(
-                        len(new_entries), instrument, aperture, file_count_threshold[0]))
+                    logging.info(('\tDark monitor skipped. MAST query has returned {} new dark files for '
+                                  '{}, {}. {} new files are required to run dark current monitor.')
+                                 .format(len(new_entries), instrument, aperture, file_count_threshold[0]))
                     monitor_run = False
 
                 # Update the query history
@@ -980,8 +993,8 @@ def stats_by_amp(self, image, amps):
             degrees_of_freedom = len(hist) - 3.
             total_pix = np.sum(hist[positive])
             p_i = gauss_fit[positive] / total_pix
-            gaussian_chi_squared[key] = (np.sum((hist[positive] - (total_pix * p_i) ** 2) / (total_pix * p_i))
-                                         / degrees_of_freedom)
+            gaussian_chi_squared[key] = (np.sum((hist[positive] - (total_pix * p_i) ** 2) / (total_pix * p_i)) /
+                                        degrees_of_freedom)
 
             # Double Gaussian fit only for full frame data (and only for
             # NIRISS, NIRCam at the moment.)


Test patch: