Repository: spacetelescope/jwql

Issue:
Recent data not showing up via instrument archive page
On the NIRSpec archive page (https://jwql.stsci.edu/nirspec/archive/#) sorting the proposals by 'Recent' results in 1491, 2418, and 2302, 1272, and 1981 as the first five entries.
Clicking down into these programs though, the exposure dates for the data are:
1491: Aug 1, 2022
2418: Feb 14, 2023
2302: Feb 3, 2023
1272: Dec 2, 2022
1981: Dec 25, 2022

But when I search using the date range page, I see that these programs do in fact have new data taken in the last day or two.  (Note that there seem to be a lot of missing thumbnails for these new files, and the preview images are almost completely full of what look like CRs https://jwql.stsci.edu/NIRSpec/jw01491027001_01202_00001_nrs2/. #1207 is open for the bad preview images)

Back on the archive page when sorted by most recent, the thumbnail link does not go to the observation level page for the new data. Neither is the new observation number listed in the drop down menu from the program/observation-level page. 

Checking the archive_database_update logs:
grep "1491 - obs:027" archive_database_update_pljwql1.stsci.edu_2023-03-07*log
grep "1491 - obs:027" archive_database_update_pljwql1.stsci.edu_2023-03-06*log
It looks like 4 root_file_info entries were created in archive_database_update_pljwql1.stsci.edu_2023-03-06-03-41.log

Not quite sure what is going on here. I'll continue looking into it.



Discussion:
I'm suspicious that a recent update to MAST may be behind this. Looks like instrument names are being changed from "<instrument>" to "<instrument>/<mode>", e.g. "MIRI" -> "MIRI/IMAGE". This is probably affecting our MAST queries.
Until all data in MAST are reprocessed using the new software, we may have to do both versions of the query in order to see all the data. https://stsci.slack.com/archives/C43BULK1T/p1678219491635759



Resolving patch:
diff --git a/jwql/utils/utils.py b/jwql/utils/utils.py
--- a/jwql/utils/utils.py
+++ b/jwql/utils/utils.py
@@ -561,8 +561,8 @@ def get_rootnames_for_instrument_proposal(instrument, proposal):
     rootnames : list
         List of rootnames for the given instrument and proposal number
     """
-    tap_service = vo.dal.TAPService("http://vao.stsci.edu/caomtap/tapservice.aspx")
-    tap_results = tap_service.search(f"select observationID from dbo.CaomObservation where collection='JWST' and maxLevel=2 and insName like '{instrument.lower()}' and prpID='{int(proposal)}'")
+    tap_service = vo.dal.TAPService("https://vao.stsci.edu/caomtap/tapservice.aspx")
+    tap_results = tap_service.search(f"select observationID from dbo.CaomObservation where collection='JWST' and maxLevel=2 and insName like '{instrument.lower()}%' and prpID='{int(proposal)}'")
     prop_table = tap_results.to_table()
     rootnames = prop_table['observationID'].data
     return rootnames.compressed()
diff --git a/jwql/website/apps/jwql/data_containers.py b/jwql/website/apps/jwql/data_containers.py
--- a/jwql/website/apps/jwql/data_containers.py
+++ b/jwql/website/apps/jwql/data_containers.py
@@ -958,8 +958,8 @@ def get_instrument_proposals(instrument):
     inst_proposals : list
         List of proposals for the given instrument
     """
-    tap_service = vo.dal.TAPService("http://vao.stsci.edu/caomtap/tapservice.aspx")
-    tap_results = tap_service.search(f"select distinct proposal_id from dbo.ObsPointing where obs_collection='JWST' and calib_level>0 and instrument_name like '{instrument.lower()}'")
+    tap_service = vo.dal.TAPService("https://vao.stsci.edu/caomtap/tapservice.aspx")
+    tap_results = tap_service.search(f"select distinct proposal_id from dbo.ObsPointing where obs_collection='JWST' and calib_level>0 and instrument_name like '{instrument.lower()}%'")
     prop_table = tap_results.to_table()
     proposals = prop_table['proposal_id'].data
     inst_proposals = sorted(proposals.compressed(), reverse=True)
@@ -1086,7 +1086,7 @@ def get_rootnames_for_proposal(proposal):
     rootnames : list
         List of rootnames for the given instrument and proposal number
     """
-    tap_service = vo.dal.TAPService("http://vao.stsci.edu/caomtap/tapservice.aspx")
+    tap_service = vo.dal.TAPService("https://vao.stsci.edu/caomtap/tapservice.aspx")
     tap_results = tap_service.search(f"select observationID from dbo.CaomObservation where collection='JWST' and maxLevel=2 and prpID='{int(proposal)}'")
     prop_table = tap_results.to_table()
     rootnames = prop_table['observationID'].data


Test patch: