Repository: spacetelescope/jwql

Issue:
Update generate_preview_images to handle different filename types
While testing `generate_preview_images.py` on the updated filesystem (#292), I notice that it chokes on certain filename types, e.g. `jw00307-o003_t001_niriss_f277w-clearp_i2d.fits` results in

```
02/27/2019 13:25:58 PM CRITICAL: Traceback (most recent call last):
  File "/home/jwqladm/repositories/jwql/jwql/utils/logging_functions.py", line 238, in wrapped
    func(*a, **kw)
  File "/home/jwqladm/repositories/jwql/jwql/utils/logging_functions.py", line 203, in wrapped
    func(*a, **kw)
  File "generate_preview_images.py", line 512, in generate_preview_images
    grouped_filenames = group_filenames(filenames)
  File "generate_preview_images.py", line 626, in group_filenames
    observation = filename_parts['observation']
KeyError: 'observation'
```

This is because this particular filename type does not have an `observation` component to it.


Discussion:
@bhilbert4 Do you know if `preview_image` supports generating preview images for `i2d` files?  Perhaps we should identify supported filetypes for `generate_preview_images` and restrict the software to only run on those.
This looks to me like a possible problem with the filename_parser. That filename you mentioned is not in the standard format.
@bhilbert4 Well, it is technically the standard format for a stage 3 product, and we have actually updated the `filename_parser` to handle these (#248), but are just now realizing that `generate_preview_images` don't support these.

I think we should be able to update `generate_preview_images.py` to handle these, but I am just wondering if `preview_images.py` would have to be heavily refactored to generate preview images for `i2d` filetypes, and if so, a larger question of if we even want to support that should probably be asked.  
`preview_images` should be set up to support 2, 3, or 4-D data, so I think it should be able to do an i2d file once it opens the file.
Do we think this bug with `generate_preview_images` is responsible for all the images that aren't currently showing up on the web app? 

For instance: https://dljwql.stsci.edu/FGS/archive/00611/ 
Hmmm, i'm not sure.  Let me dig into it.
After some digging, I think the error @laurenmarietta points out is because the updated filesystem contains some programs that the old filesystem did not (`00611` being one of them) and so preview images have yet to be created for those.

I suspect that once we solve this issue (#309) and generate updated preview images, this problem will go away. 
@bhilbert4 I might need your help with implementing changes to `generate_preview_images` to support stage 3 filenames.  It appears that it is not going to be that straightforward.  For instance, some of the code is dependent on the existence of the `detector` in the filenames, but stage 3 filenames do not have a `detector` specified.  We will have to figure out how we want to group such files and make them available in the `view_image` webpages.  

Am I correct in understating that a stage 3 data product may be a product of several individual observations/rootnames?  Or is there still 1-to-1 correspondence? 
Yes, I think you're right on that. Multiple observations can be combined in stage 3. I think in that case the default is to have the association name in some form within the filename.
Hopefully there are enough details in the pipeline/DMS documentation that we can figure out all the options.
One option may be to grab all the association files (json format) produced by DMS and examine them. Within an association file there is a field called `name` that contains the output name (minus suffix). Then we can search for files with that base name.

For example:
```
   "name": "jw96090-o001_t001_nircam_f322w2-f323n"
```

https://jwst-docs.stsci.edu/display/JDAT/Understanding+Associations

Resolving patch:
diff --git a/jwql/jwql_monitors/generate_preview_images.py b/jwql/jwql_monitors/generate_preview_images.py
--- a/jwql/jwql_monitors/generate_preview_images.py
+++ b/jwql/jwql_monitors/generate_preview_images.py
@@ -26,8 +26,9 @@
         python generate_preview_images.py
 """
 
-from glob import glob
+import glob
 import logging
+import multiprocessing
 import os
 import re
 
@@ -215,7 +216,7 @@ def check_existence(file_list, outdir):
                         file_parts['parallel_seq_id'], file_parts['activity'],
                         file_parts['exposure_id'], mosaic_str, file_parts['suffix'])
 
-    current_files = glob(os.path.join(outdir, search_string))
+    current_files = glob.glob(os.path.join(outdir, search_string))
     if len(current_files) > 0:
         return True
     else:
@@ -496,35 +497,151 @@ def find_data_channel(detectors):
     return channel
 
 
+def get_base_output_name(filename_dict):
+    """Returns the base output name used for preview images and
+    thumbnails.
+
+    Parameters
+    ----------
+    filename_dict : dict
+        A dictionary containing parsed filename parts via
+        ``filename_parser``
+
+    Returns
+    -------
+    base_output_name : str
+        The base output name, e.g. ``jw96090001002_03101_00001_nrca2_rate``
+    """
+
+    base_output_name = 'jw{}{}{}_{}{}{}_{}_'.format(
+        filename_dict['program_id'], filename_dict['observation'],
+        filename_dict['visit'], filename_dict['visit_group'],
+        filename_dict['parallel_seq_id'], filename_dict['activity'],
+        filename_dict['exposure_id'])
+
+    return base_output_name
+
+
 @log_fail
 @log_info
 def generate_preview_images():
-    """The main function of the ``generate_preview_image`` module."""
+    """The main function of the ``generate_preview_image`` module.
+    See module docstring for further details."""
 
     # Begin logging
     logging.info("Beginning the script run")
 
-    filesystem = get_config()['filesystem']
-    preview_image_filesystem = get_config()['preview_image_filesystem']
-    thumbnail_filesystem = get_config()['thumbnail_filesystem']
+    # Process programs in parallel
+    program_list = [os.path.basename(item) for item in glob.glob(os.path.join(get_config()['filesystem'], '*'))]
+    pool = multiprocessing.Pool(processes=int(get_config()['cores']))
+    pool.map(process_program, program_list)
+    pool.close()
+    pool.join()
+
+    # Complete logging:
+    logging.info("Completed.")
+
+
+def group_filenames(filenames):
+    """Given a list of JWST filenames, group together files from the
+    same exposure. These files will share the same ``program_id``,
+    ``observation``, ``visit``, ``visit_group``, ``parallel_seq_id``,
+    ``activity``, ``exposure``, and ``suffix``. Only the ``detector``
+    will be different. Currently only NIRCam files for a given exposure
+    will be grouped together. For other instruments multiple files for
+    a given exposure will be kept separate from one another and no
+    mosaic will be made.  Stage 3 files will remain as individual
+    files, and will not be grouped together with any other files.
+
+    Parameters
+    ----------
+    filenames : list
+        list of filenames
+
+    Returns
+    -------
+    grouped : list
+        grouped list of filenames where each element is a list and
+        contains the names of filenames with matching exposure
+        information.
+    """
+
+    # Some initializations
+    grouped, matched_names = [], []
+    filenames.sort()
 
-    filenames = glob(os.path.join(filesystem, '*/*.fits'))
+    # Loop over each file in the list of good files
+    for filename in filenames:
+
+        # Holds list of matching files for exposure
+        subgroup = []
+
+        # Generate string to be matched with other filenames
+        filename_dict = filename_parser(os.path.basename(filename))
+
+        # If the filename was already involved in a match, then skip
+        if filename not in matched_names:
+
+            # For stage 3 filenames, treat individually
+            if 'stage_3' in filename_dict['filename_type']:
+                matched_names.append(filename)
+                subgroup.append(filename)
+
+            # Group together stage 1 and 2 filenames
+            elif filename_dict['filename_type'] == 'stage_1_and_2':
+
+                # Determine detector naming convention
+                if filename_dict['detector'].upper() in NIRCAM_SHORTWAVE_DETECTORS:
+                    detector_str = 'NRC[AB][1234]'
+                elif filename_dict['detector'].upper() in NIRCAM_LONGWAVE_DETECTORS:
+                    detector_str = 'NRC[AB]5'
+                else:  # non-NIRCam detectors
+                    detector_str = filename_dict['detector'].upper()
+
+                # Build pattern to match against
+                base_output_name = get_base_output_name(filename_dict)
+                match_str = '{}{}_{}.fits'.format(base_output_name, detector_str, filename_dict['suffix'])
+                match_str = os.path.join(os.path.dirname(filename), match_str)
+                pattern = re.compile(match_str, re.IGNORECASE)
+
+                # Try to match the substring to each good file
+                for file_to_match in filenames:
+                    if pattern.match(file_to_match) is not None:
+                        matched_names.append(file_to_match)
+                        subgroup.append(file_to_match)
+
+        if len(subgroup) > 0:
+            grouped.append(subgroup)
+
+    return grouped
+
+
+def process_program(program):
+    """Generate preview images and thumbnails for the given program.
+
+    Parameters
+    ----------
+    program : str
+        The program identifier (e.g. ``88600``)
+    """
+
+    # Group together common exposures
+    filenames = glob.glob(os.path.join(get_config()['filesystem'], program, '*.fits'))
     grouped_filenames = group_filenames(filenames)
     logging.info('Found {} filenames'.format(len(filenames)))
 
     for file_list in grouped_filenames:
         filename = file_list[0]
+
         # Determine the save location
         try:
             identifier = 'jw{}'.format(filename_parser(filename)['program_id'])
-        except ValueError as error:
+        except ValueError:
             identifier = os.path.basename(filename).split('.fits')[0]
+        preview_output_directory = os.path.join(get_config()['preview_image_filesystem'], identifier)
+        thumbnail_output_directory = os.path.join(get_config()['thumbnail_filesystem'], identifier)
 
-        preview_output_directory = os.path.join(preview_image_filesystem, identifier)
-        thumbnail_output_directory = os.path.join(thumbnail_filesystem, identifier)
-
-        # Check to see if the preview images already exist and skip
-        # if they do
+        # Check to see if the preview images already exist and skip if they do
         file_exists = check_existence(file_list, preview_output_directory)
         if file_exists:
             logging.info("JPG already exists for {}, skipping.".format(filename))
@@ -544,7 +661,7 @@ def generate_preview_images():
         # than one detector was used), then create a mosaic
         max_size = 8
         numfiles = len(file_list)
-        if numfiles != 1:
+        if numfiles > 1:
             try:
                 mosaic_image, mosaic_dq = create_mosaic(file_list)
                 logging.info('Created mosiac for:')
@@ -578,100 +695,10 @@ def generate_preview_images():
                 im.file = dummy_file
 
             im.make_image(max_img_size=max_size)
+            logging.info('Created preview image and thumbnail for: {}'.format(filename))
         except ValueError as error:
             logging.warning(error)
 
-    # Complete logging:
-    logging.info("Completed.")
-
-
-def group_filenames(input_files):
-    """Given a list of JWST filenames, group together files from the
-    same exposure. These files will share the same ``program_id``,
-    ``observation``, ``visit``, ``visit_group``, ``parallel_seq_id``,
-    ``activity``, ``exposure``, and ``suffix``. Only the ``detector``
-    will be different. Currently only NIRCam files for a given exposure
-    will be grouped together. For other instruments multiple files for
-    a given exposure will be kept separate from one another and no
-    mosaic will be made.
-
-    Parameters
-    ----------
-    input_files : list
-        list of filenames
-
-    Returns
-    -------
-    grouped : list
-        grouped list of filenames where each element is a list and
-        contains the names of filenames with matching exposure
-        information.
-    """
-
-    grouped = []
-
-    # Sort files first
-    input_files.sort()
-
-    goodindex = np.arange(len(input_files))
-    input_files = np.array(input_files)
-
-    # Loop over each file in the list of good files
-    for index, full_filename in enumerate(input_files[goodindex]):
-        file_directory, filename = os.path.split(full_filename)
-
-        # Generate string to be matched with other filenames
-        filename_parts = filename_parser(filename)
-        program = filename_parts['program_id']
-        observation = filename_parts['observation']
-        visit = filename_parts['visit']
-        visit_group = filename_parts['visit_group']
-        parallel = filename_parts['parallel_seq_id']
-        activity = filename_parts['activity']
-        exposure = filename_parts['exposure_id']
-        detector = filename_parts['detector'].upper()
-        suffix = filename_parts['suffix']
-
-        observation_base = 'jw{}{}{}_{}{}{}_{}_'.format(
-            program, observation, visit, visit_group,
-            parallel, activity, exposure)
-
-        if detector in NIRCAM_SHORTWAVE_DETECTORS:
-            detector_str = 'NRC[AB][1234]'
-        elif detector in NIRCAM_LONGWAVE_DETECTORS:
-            detector_str = 'NRC[AB]5'
-        else:  # non-NIRCam detectors - should never be used I think??
-            detector_str = detector
-        match_str = '{}{}_{}.fits'.format(observation_base, detector_str, suffix)
-        match_str = os.path.join(file_directory, match_str)
-        pattern = re.compile(match_str, re.IGNORECASE)
-
-        # Try to match the substring to each good file
-        matches = []
-        matched_name = []
-        for index2, file2match in enumerate(input_files[goodindex]):
-            match = pattern.match(file2match)
-
-            # Add any files that match the string
-            if match is not None:
-                matched_name.append(file2match)
-                matches.append(goodindex[index2])
-        # For any matched files, remove from goodindex so we don't
-        # use them as a basis for matching later
-        all_locs = []
-        for num in matches:
-            loc = np.where(goodindex == num)
-            all_locs.append(loc[0][0])
-        if len(all_locs) != 0:
-            # Delete matched file indexes from the list of
-            # files to search
-            goodindex = np.delete(goodindex, all_locs)
-
-            # Add the list of matched files to the overall list of files
-            grouped.append(matched_name)
-
-    return grouped
-
 
 if __name__ == '__main__':
 
diff --git a/jwql/utils/preview_image.py b/jwql/utils/preview_image.py
--- a/jwql/utils/preview_image.py
+++ b/jwql/utils/preview_image.py
@@ -50,7 +50,7 @@
 import matplotlib.colors as colors
 
 # Only import jwst if not running from readthedocs
-if 'build' and 'project' and 'jwql' not in socket.gethostname():
+if 'build' and 'project' not in socket.gethostname():
     from jwst.datamodels import dqflags
 
 
@@ -165,7 +165,7 @@ def find_limits(self, data, pixmap, clipperc):
         numclip = np.int(clipperc * nelem)
         sorted = np.sort(data[pixmap], axis=None)
         minval = sorted[numclip]
-        maxval = sorted[-numclip-1]
+        maxval = sorted[-numclip - 1]
         return (minval, maxval)
 
     def get_data(self, filename, ext):
@@ -212,13 +212,18 @@ def get_data(self, filename, ext):
                     yd, xd = data.shape[-2:]
                     dq = np.ones((yd, xd), dtype="bool")
 
+
                 # Collect information on aperture location within the
                 # full detector. This is needed for mosaicking NIRCam
                 # detectors later.
-                self.xstart = hdulist[0].header['SUBSTRT1']
-                self.ystart = hdulist[0].header['SUBSTRT2']
-                self.xlen = hdulist[0].header['SUBSIZE1']
-                self.ylen = hdulist[0].header['SUBSIZE2']
+                try:
+                    self.xstart = hdulist[0].header['SUBSTRT1']
+                    self.ystart = hdulist[0].header['SUBSTRT2']
+                    self.xlen = hdulist[0].header['SUBSIZE1']
+                    self.ylen = hdulist[0].header['SUBSIZE2']
+                except KeyError:
+                    logging.warning('SUBSTR and SUBSIZE header keywords not found')
+
         else:
             raise FileNotFoundError(('WARNING: {} does not exist!'.format(filename)))
 
@@ -322,16 +327,15 @@ def make_figure(self, image, integration_number, min_value, max_value,
                 tlabelstr = [format_string % number for number in tlabelflt]
                 cbar = fig.colorbar(cax, ticks=tickvals)
                 cbar.ax.set_yticklabels(tlabelstr)
-                cbar.ax.tick_params(labelsize=maxsize * 5./4)
-                # cbar.ax.set_ylabel('Signal', rotation=270, fontsize=maxsize*5./4)
-                ax.set_xlabel('Pixels', fontsize=maxsize * 5./4)
-                ax.set_ylabel('Pixels', fontsize=maxsize * 5./4)
+                cbar.ax.tick_params(labelsize=maxsize * 5. / 4)
+                ax.set_xlabel('Pixels', fontsize=maxsize * 5. / 4)
+                ax.set_ylabel('Pixels', fontsize=maxsize * 5. / 4)
                 ax.tick_params(labelsize=maxsize)
                 plt.rcParams.update({'axes.titlesize': 'small'})
-                plt.rcParams.update({'font.size': maxsize * 5./4})
-                plt.rcParams.update({'axes.labelsize': maxsize * 5./4})
-                plt.rcParams.update({'ytick.labelsize': maxsize * 5./4})
-                plt.rcParams.update({'xtick.labelsize': maxsize * 5./4})
+                plt.rcParams.update({'font.size': maxsize * 5. / 4})
+                plt.rcParams.update({'axes.labelsize': maxsize * 5. / 4})
+                plt.rcParams.update({'ytick.labelsize': maxsize * 5. / 4})
+                plt.rcParams.update({'xtick.labelsize': maxsize * 5. / 4})
 
         elif scale == 'linear':
             fig, ax = plt.subplots(figsize=(xsize, ysize))


Test patch: