Repository: spacetelescope/jwql

Issue:
Better way to get a list of instrument proposals and categories
_Issue [JWQL-162](https://jira.stsci.edu/browse/JWQL-162) was created on JIRA by [Bryan Hilbert](https://jira.stsci.edu/secure/ViewProfile.jspa?name=hilbert):_

In data_containers.get_proposals_by_category(), the code retrieves a list of proposals and associated categories for a given instrument. It does this in a very inefficient way, by querying MAST for all exposures for a given instrument, and then picking out the unique proposal names. But for NIRCam, we are currently bumping up against our (already bumped up) MAST query limit of 500,000 results. (NIRCam currently returns 540,000 results.) Missing results are leading to missing proposals in the list, which causes problems downstream. Related to JWQL-160, we have bumped up the MAST query limit in order to solve the problem in the short term. But in the long term, we need a better way to get a list of the proposals and categories for a given instrument. 

I have asked Dick Shaw, as part of [Help Desk ticket INC0196521](https://stsci.service-now.com/stars?id=ticket&table=incident&sys_id=3448820947ef79d0289c4f32736d4309) if there is a better way to get this information.


Discussion:
_Comment by [Bryan Hilbert](https://jira.stsci.edu/secure/ViewProfile.jspa?name=hilbert) on [JIRA](https://jira.stsci.edu/browse/JWQL-162?focusedCommentId=679120#comment-679120):_

From David Rodriguez:

Re: your last question as to whether you can get both the category and program efficiently, your original TAP query can be modified to include the prpProject field which includes that information. 
For example: select distinct prpID, prpProject from CaomObservation where collection='JWST' and maxLevel>0 and insName like 'nircam%'
That will get you the unique program and categories for all nircam observations. I'm not aware if TAP queries have any limits, but by using distinct you should be getting a relatively small number of rows (about 300).

_Comment by [Bryan Hilbert](https://jira.stsci.edu/secure/ViewProfile.jspa?name=hilbert) on [JIRA](https://jira.stsci.edu/browse/JWQL-162?focusedCommentId=681313#comment-681313):_

git is not working for me at the moment due to a recent OS upgrade, but once that is working, starting with line 1259 in data_containers.py, we can do:

tap_results = tap_service.search(f"""select distinct prpID,prpProject from CaomObservation where collection='JWST' and maxLevel>0 and insName like '{instrument.lower()}%'""")
prop_table = tap_results.to_table()
proposals = prop_table['prpID'].data
proposal_category = prop_table['prpProject'].data
Tests show that the pyvo results are in order of increasing proposal number.
proposal_list = list(proposals.data)
category_list = list(proposal_category.data)
proposal_list, category_list = zip(*sorted(zip(proposal_list, category_list), reverse=True))

Don't forget to either put this in a new function, or update the calls to get_instrument_proposals() such that it can return a list of proposal numbers and a list of proposal categories.


Resolving patch:
diff --git a/jwql/website/apps/jwql/data_containers.py b/jwql/website/apps/jwql/data_containers.py
--- a/jwql/website/apps/jwql/data_containers.py
+++ b/jwql/website/apps/jwql/data_containers.py
@@ -1391,7 +1391,7 @@ def get_preview_images_by_rootname(rootname):
 
 
 def get_proposals_by_category(instrument):
-    """Return a dictionary of program numbers based on category type
+    """Return a dictionary of program numbers and category type
     Parameters
     ----------
     instrument : str
@@ -1400,24 +1400,16 @@ def get_proposals_by_category(instrument):
     Returns
     -------
     category_sorted_dict : dict
-        Dictionary with category as the key and a list of program id's as the value
+        Dictionary with program number as the key and program category as the value
     """
+    tap_service = vo.dal.TAPService("https://vao.stsci.edu/caomtap/tapservice.aspx")
+    tap_results = tap_service.search(f"""select distinct prpID,prpProject from CaomObservation where collection='JWST'
+                                     and maxLevel>0 and insName like '{instrument.lower()}%'""")
+    # Put the results into an astropy Table
+    prop_table = tap_results.to_table()
 
-    service = "Mast.Jwst.Filtered.{}".format(instrument)
-    params = {"columns": "program, category",
-              "filters": [{'paramName': 'instrume', 'values': [instrument]}]}
-    response = Mast.service_request_async(service, params)
-    results = response[0].json()['data']
-
-    if len(results) == MAST_QUERY_LIMIT:
-        logging.error(f"MAST_QUERY_LIMIT of {MAST_QUERY_LIMIT} reached for {instrument} in get_proposals_by_category")
-
-    # Get all unique dictionaries
-    unique_results = list(map(dict, set(tuple(sorted(sub.items())) for sub in results)))
-
-    # Make a dictionary of {program: category} to pull from
-    proposals_by_category = {d['program']: d['category'] for d in unique_results}
-
+    # Convert to a dictionary
+    proposals_by_category = {int(d['prpID']): d['prpProject'] for d in prop_table}
     return proposals_by_category
 
 


Test patch: