Repository: spacetelescope/jwql

Issue:
Expand tests for api_views
As @frboffi and @catherine-martlin suggested, it would be a good idea to expand the tests in `test_api_views` to include each of the five JWST instruments.  Currently, we just have tests for various data from FGS.  


Discussion:
Heads up that this might end up being a bit tricky - it looks like every instrument except FGS has a file in the filesystem with funky formatting that is causing `filename_parser` to throw an error. Some examples:
- jw00733001001_02101_00004-**seg001_nis**_uncal.fits
- jw00733003001_02101_00002-**seg001_nrs1**_rate.fits
- jw94015001002_02104_00001_nrcblong_**o001**_crf.fits
- jw**80600-o009_t001**\_miri\_**f1130w**_i2d.fits (!!?!)
@laurenmarietta Thanks for digging up those examples.  I think I will have to ask some of the archive folks why there are files in the filesystem that don't conform to the naming convention.  Perhaps the copy of the filesystem that we have been using is now outdated and the issue has since been fixed. 
Those seg001 files are Time series observations. The TS observations can be very large, so these will be segmented in files of < 2Gb so these can be calibrated and distributed.
The last example is the product of an association created at Stage 3 of the calibration pipeline.
@rizeladiaz Thanks for the notes.  From what you describe, my understanding is that these naming conventions are indeed correct and expected?  Do you know if there exists any documentation anywhere that describes the full allowable JWST naming conventions?  

Currently, we are going by this: https://innerspace.stsci.edu/display/~donald/JWST+File+Names+and+Locations.  However, this document does not describe the naming convention for segmented time series observations or the stage 3 association products.
The official sources of information (latter to be merged) are:

https://jwst-docs.stsci.edu/display/JDAT/File+Naming+Conventions+and+Data+Products
and
https://jwst-pipeline.readthedocs.io/en/stable/jwst/introduction.html#output-file-and-associations

note, however, that the naming convention for the TS files was decided recently and it is not in the current documentation. One of the examples mentioned here is stage3. Final versions of all the Stage 3 exposures and extensions won't be final until we get more test data into the system; i.e. once DMS starts getting data that can go all the way to stage 3, they might find there are some changes needed. 
Thanks for the information @rizeladiaz! It is quite helpful!

Resolving patch:
diff --git a/jwql/website/apps/jwql/data_containers.py b/jwql/website/apps/jwql/data_containers.py
--- a/jwql/website/apps/jwql/data_containers.py
+++ b/jwql/website/apps/jwql/data_containers.py
@@ -421,13 +421,11 @@ def get_instrument_proposals(instrument):
     """
 
     service = "Mast.Jwst.Filtered.{}".format(instrument)
-    params = {"columns": "filename",
+    params = {"columns": "program",
               "filters": []}
     response = Mast.service_request_async(service, params)
     results = response[0].json()['data']
-
-    filenames = [result['filename'] for result in results]
-    proposals = list(set(filename_parser(filename)['program_id'] for filename in filenames))
+    proposals = list(set(result['program'] for result in results))
 
     return proposals
 
@@ -461,17 +459,11 @@ def get_preview_images_by_instrument(inst):
     # Parse the results to get the rootnames
     filenames = [result['filename'].split('.')[0] for result in results]
 
-    # Build list of available preview images
-    preview_images = []
-    for filename in filenames:
-        proposal = filename_parser(filename)['program_id']
-        preview_images.extend(glob.glob(os.path.join(
-            PREVIEW_IMAGE_FILESYSTEM,
-            'jw{}'.format(proposal),
-            '{}*.jpg'.format(filename))))
+    # Get list of all preview_images
+    preview_images = glob.glob(os.path.join(PREVIEW_IMAGE_FILESYSTEM, '*', '*.jpg'))
 
-    # Only return the filenames
-    preview_images = [os.path.basename(preview_image) for preview_image in preview_images]
+    # Get subset of preview images that match the filenames
+    preview_images = [item for item in preview_images if os.path.basename(item).split('_integ')[0] in filenames]
 
     return preview_images
 
@@ -598,17 +590,11 @@ def get_thumbnails_by_instrument(inst):
     # Parse the results to get the rootnames
     filenames = [result['filename'].split('.')[0] for result in results]
 
-    # Build list of available preview images
-    thumbnails = []
-    for filename in filenames:
-        proposal = filename_parser(filename)['program_id']
-        thumbnails.extend(glob.glob(os.path.join(
-            THUMBNAIL_FILESYSTEM,
-            'jw{}'.format(proposal),
-            '{}*.thumb'.format(filename))))
+    # Get list of all thumbnails
+    thumbnails = glob.glob(os.path.join(THUMBNAIL_FILESYSTEM, '*', '*.thumb'))
 
-    # Only return the filenames
-    thumbnails = [os.path.basename(thumbnail) for thumbnail in thumbnails]
+    # Get subset of preview images that match the filenames
+    thumbnails = [item for item in thumbnails if os.path.basename(item).split('_integ')[0] in filenames]
 
     return thumbnails
 


Test patch:
diff --git a/jwql/tests/test_api_views.py b/jwql/tests/test_api_views.py
--- a/jwql/tests/test_api_views.py
+++ b/jwql/tests/test_api_views.py
@@ -23,19 +23,40 @@
 import urllib.request
 
 from jwql.utils.utils import get_base_url
-
-
-urls = [
-    'api/proposals/',  # all_proposals
-    'api/86700/filenames/',  # filenames_by_proposal
-    'api/jw86700005001_02101_00001_guider1/filenames/',  # filenames_by_rootname
-    'api/fgs/proposals/',  # instrument_proposals
-    'api/fgs/preview_images/',  # preview_images_by_instrument
-    'api/86700/preview_images/',  # preview_images_by_proposal
-    'api/jw86700005001_02101_00001_guider1/preview_images/',  # preview_images_by_rootname
-    'api/fgs/thumbnails/',  # thumbnails_by_instrument
-    'api/86700/thumbnails/',  # thumbnails_by_proposal
-    'api/jw86700005001_02101_00001_guider1/thumbnails/']  # thumbnails_by_rootname
+from jwql.utils.constants import JWST_INSTRUMENT_NAMES
+
+urls = []
+
+# Generic URLs
+urls.append('api/proposals/')  # all_proposals
+
+# Instrument-specific URLs
+for instrument in JWST_INSTRUMENT_NAMES:
+    urls.append('api/{}/proposals/'.format(instrument))  # instrument_proposals
+    urls.append('api/{}/preview_images/'.format(instrument))  # preview_images_by_instrument
+    urls.append('api/{}/thumbnails/'.format(instrument))  # thumbnails_by_instrument
+
+# Proposal-specific URLs
+proposals = ['86700',  # FGS
+             '98012',  # MIRI
+             '93025',  # NIRCam
+             '00308',  # NIRISS
+             '96213']  # NIRSpec
+for proposal in proposals:
+    urls.append('api/{}/filenames/'.format(proposal))  # filenames_by_proposal
+    urls.append('api/{}/preview_images/'.format(proposal))  # preview_images_by_proposal
+    urls.append('api/{}/thumbnails/'.format(proposal))  # thumbnails_by_proposal
+
+# Filename-specific URLs
+rootnames = ['jw86600007001_02101_00001_guider2',  # FGS
+             'jw98012001001_02102_00001_mirimage',  # MIRI
+             'jw93025001001_02102_00001_nrca2',  # NIRCam
+             'jw00308001001_02101_00001_nis',  # NIRISS
+             'jw96213001001_02101_00001_nrs1']  # NIRSpec
+for rootname in rootnames:
+    urls.append('api/{}/filenames/'.format(rootname))  # filenames_by_rootname
+    urls.append('api/{}/preview_images/'.format(rootname))  # preview_images_by_rootname
+    urls.append('api/{}/thumbnails/'.format(rootname))  # thumbnails_by_rootname
 
 
 @pytest.mark.xfail
@@ -53,6 +74,7 @@ def test_api_views(url):
     # Build full URL
     base_url = get_base_url()
     url = '{}/{}'.format(base_url, url)
+    print('Testing {}'.format(url))
 
     # Determine the type of data to check for based on the url
     data_type = url.split('/')[-2]