Repository: spacetelescope/jwst

Issue:
Combine S200A1 and S200A2 spectra in level 3 products
_Issue [JP-3233](https://jira.stsci.edu/browse/JP-3233) was created on JIRA by [Jeff Valenti](https://jira.stsci.edu/secure/ViewProfile.jspa?name=valenti):_

{*}Request{*}: For fixed slit spectroscopy with NIRSpec H gratings, enhance the association generator so that spec3 will combine exposures with the target in the S200A1 and S200A2 slits. From a data quality perspective, this will double the number of exposures being combined, substantially improving outlier rejection. From a usability perspective, the user will get one product with complete wavelength coverage, rather than two products each with a detector gap in a different wavelength range. Depending on the implementation, spec2 may be changed to encode in SOURCEID the slit containing the target and the slit being extracted. This would help users identify which level 3 products contain the target and may simplify association logic at level 3.

{*}Context{*}: For fixed slit spectroscopy with NIRSpec H gratings, [JDox](https://jwst-docs.stsci.edu/jwst-near-infrared-spectrograph/nirspec-observing-strategies/nirspec-dithering-recommended-strategies#NIRSpecDitheringRecommendedStrategies-NIRSpecdecisiontreeinfixedslitsmode) recommends taking exposures with the target in S200A1 and S200A2, if the observer wants complete wavelength coverage. [APT](https://jwst-docs.stsci.edu/jppom/nirspec-templates/nirspec-fixed-slit-spectroscopy-template-parameters#NIRSpecFixedSlitSpectroscopyTemplateParameters-Slit) provides an "S200A1 and S200A2" option for this purpose.

Discussion:
hbushouse: Reviewers: In order to incorporate the changes in ASN rules, I felt it best to no longer include the "fxd_slit" info in the "opt_elem" info, but rather keep them separate. So now "opt_elem" only includes filter and pupil wheel info, while any NRS FS slit names are handled in a separate constraint. I've also created a new "dms_product_name_nrsfs_sources" function, based on the existing "dms_product_name_sources", that works with both "source_id" and "slit_name" in the output products. This is only used for NRS FS mode. All other multi-source modes (e.g. NIRCam and NIRISS WFSS, NIRSpec MOS) continue to use the existing "dms_product_name_sources" routine, which only works with "source_id" in the product names.

All of these changes lead to LOTS of differences in regression tests, due to the new product name template syntax contained in spec3 ASN files for NRS FS data, the changes in actual output file names for all NRS FS tests, and of course the changes in "source_id" values both within the datasets themselves and in the output file names.
hbushouse: Initial regression test run started at https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/913  This will likely take several iterations to get a run that doesn't crash, because input data to some of the tests will likely need updating to conform to the new expectations of source_id assignments during level-2b processing of NRS FS exposures.
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/spacetelescope/jwst/pull/7879?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) Report
Patch coverage is **`70.37%`** of modified lines.

| [Files Changed](https://app.codecov.io/gh/spacetelescope/jwst/pull/7879?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) | Coverage |
|---|---|
| [jwst/associations/lib/rules\_level2\_base.py](https://app.codecov.io/gh/spacetelescope/jwst/pull/7879?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope#diff-andzdC9hc3NvY2lhdGlvbnMvbGliL3J1bGVzX2xldmVsMl9iYXNlLnB5) | `ø` |
| [jwst/pipeline/calwebb\_spec3.py](https://app.codecov.io/gh/spacetelescope/jwst/pull/7879?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope#diff-andzdC9waXBlbGluZS9jYWx3ZWJiX3NwZWMzLnB5) | `7.69%` |
| [jwst/associations/lib/dms\_base.py](https://app.codecov.io/gh/spacetelescope/jwst/pull/7879?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope#diff-andzdC9hc3NvY2lhdGlvbnMvbGliL2Rtc19iYXNlLnB5) | `82.35%` |
| [jwst/assign\_wcs/nirspec.py](https://app.codecov.io/gh/spacetelescope/jwst/pull/7879?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope#diff-andzdC9hc3NpZ25fd2NzL25pcnNwZWMucHk=) | `90.90%` |
| [jwst/associations/lib/rules\_level3.py](https://app.codecov.io/gh/spacetelescope/jwst/pull/7879?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope#diff-andzdC9hc3NvY2lhdGlvbnMvbGliL3J1bGVzX2xldmVsMy5weQ==) | `100.00%` |
| [jwst/associations/lib/rules\_level3\_base.py](https://app.codecov.io/gh/spacetelescope/jwst/pull/7879?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope#diff-andzdC9hc3NvY2lhdGlvbnMvbGliL3J1bGVzX2xldmVsM19iYXNlLnB5) | `100.00%` |


:loudspeaker: Thoughts on this report? [Let us know!](https://about.codecov.io/pull-request-comment-report/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope).

hbushouse: All CI failures are now down to unrelated mismatches between stcal/main and stcal released (CI tests use stcal release).
hbushouse: Latest regression test run results are at https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/915/testReport/ and it all looks good. Lots of expected changes in asn product name results, as well as source_id values in calibrated data products. All truths will be updated after this gets merged. Declaring this ready for review.

Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index 049c4448e..061eaf313 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -9,6 +9,11 @@ assign_wcs
 - Save bounding box to imaging WCS matching the shape of the data, for datamodels
   without a defined bounding box. [#7809]
 
+- Update the assignment of "source_id" values for NIRSpec fixed-slit exposures, so
+  that the slit containing the primary target always has source_id=1 and secondary
+  slits use a two-digit source_id value that reflects both the primary target in
+  use and the secondary slit from which the data are extracted. [#7879]
+
 associations
 ------------
 
@@ -23,6 +28,17 @@ associations
 
 - Remove order dependency on association diffing. [#7853]
 
+- Update the Level 3 association rules for NIRSpec fixed-slit so that observations
+  the put the primary target in both the S200A1 and S200A2 slits get the data from
+  those two slits combined into a single final product. [#7879]
+
+- Update the Level 3 product name construction for NIRSpec fixed-slit observations
+  so that both the "source_id" and "slit_name" are left as variables for the
+  "calwebb_spec3" to populate at execution time. This update required an update to
+  the handling of the "opt_elem" attribute, so that it now only contains filter
+  and pupil information, while slit information is contained in the separate
+  attribute "fxd_slit". [#7879]
+
 background
 ----------
 
@@ -35,6 +51,14 @@ calwebb_spec2
   that is returned by the pipeline to ensure a file is created with the
   expected ``_cal`` suffix. [#7772]
 
+calwebb_spec3
+-------------
+
+- Updated to create output product names for NIRSpec fixed-slit data based on
+  both the "source_id" and "slit_name" values for each set of slit data, so that
+  the product name properly reflects the slit from which the data were taken.
+  [#7879]
+
 charge_migration
 ----------------
 
diff --git a/jwst/assign_wcs/nirspec.py b/jwst/assign_wcs/nirspec.py
index 39fb4eba7..532ea0766 100644
--- a/jwst/assign_wcs/nirspec.py
+++ b/jwst/assign_wcs/nirspec.py
@@ -413,7 +413,7 @@ def get_open_slits(input_model, reference_files=None, slit_y_range=[-.55, .55]):
     elif exp_type == "nrs_fixedslit":
         slits = get_open_fixed_slits(input_model, slit_y_range)
     elif exp_type == "nrs_brightobj":
-        slits = [Slit('S1600A1', 3, 0, 0, 0, slit_y_range[0], slit_y_range[1], 5, 4)]
+        slits = [Slit('S1600A1', 3, 0, 0, 0, slit_y_range[0], slit_y_range[1], 5, 1)]
     elif exp_type in ["nrs_lamp", "nrs_autowave"]:
         if lamp_mode in ['fixedslit', 'brightobj']:
             slits = get_open_fixed_slits(input_model, slit_y_range)
@@ -434,17 +434,37 @@ def get_open_fixed_slits(input_model, slit_y_range=[-.55, .55]):
     """ Return the opened fixed slits."""
     if input_model.meta.subarray.name is None:
         raise ValueError("Input file is missing SUBARRAY value/keyword.")
+    if input_model.meta.instrument.fixed_slit is None:
+        input_model.meta.instrument.fixed_slit = 'NONE'
 
-    slits = []
+    slit_nums = {'NONE':0, 'S200A1':1, 'S200A2':2, 'S400A1':3, 'S1600A1':4, 'S200B1':5}
+    primary_slit = input_model.meta.instrument.fixed_slit
     ylow, yhigh = slit_y_range
 
-    s2a1 = Slit('S200A1', 0, 0, 0, 0, ylow, yhigh, 5, 1)
-    s2a2 = Slit('S200A2', 1, 0, 0, 0, ylow, yhigh, 5, 2)
-    s4a1 = Slit('S400A1', 2, 0, 0, 0, ylow, yhigh, 5, 3)
-    s16a1 = Slit('S1600A1', 3, 0, 0, 0, ylow, yhigh, 5, 4)
-    s2b1 = Slit('S200B1', 4, 0, 0, 0, ylow, yhigh, 5, 5)
-
+    # Slits are defined with hardwired source ID's, based on the assignments
+    # in the "slit_nums" dictionary. Exact assignments depend on whether the
+    # slit is the "primary" and hence contains the target of interest. The
+    # source_id for the primary slit is always 1, while source_ids for secondary
+    # slits is a two-digit value, where the first (tens) digit corresponds to
+    # primary slit in use and the second (ones) digit corresponds to the
+    # secondary slit number. All fixed slits are assigned to the virtual MSA
+    # quadrant 5.
+    #
+    # Slit(Name, ShutterID, DitherPos, Xcen, Ycen, Ymin, Ymax, Quad, SourceID)
+    s2a1 = Slit('S200A1', 0, 0, 0, 0, ylow, yhigh, 5,
+        1 if primary_slit=='S200A1' else 10*slit_nums[primary_slit] + 1)
+    s2a2 = Slit('S200A2', 1, 0, 0, 0, ylow, yhigh, 5,
+        1 if primary_slit=='S200A2' else 10*slit_nums[primary_slit] + 2)
+    s4a1 = Slit('S400A1', 2, 0, 0, 0, ylow, yhigh, 5,
+        1 if primary_slit=='S400A1' else 10*slit_nums[primary_slit] + 3)
+    s16a1 = Slit('S1600A1', 3, 0, 0, 0, ylow, yhigh, 5,
+        1 if primary_slit=='S1600A1' else 10*slit_nums[primary_slit] + 4)
+    s2b1 = Slit('S200B1', 4, 0, 0, 0, ylow, yhigh, 5,
+        1 if primary_slit=='S200B1' else 10*slit_nums[primary_slit] + 5)
+
+    # Decide which slits need to be added to this exposure
     subarray = input_model.meta.subarray.name.upper()
+    slits = []
     if subarray == "SUBS200A1":
         slits.append(s2a1)
     elif subarray == "SUBS200A2":
diff --git a/jwst/associations/lib/dms_base.py b/jwst/associations/lib/dms_base.py
index 7d4bfd213..bbd8053a6 100644
--- a/jwst/associations/lib/dms_base.py
+++ b/jwst/associations/lib/dms_base.py
@@ -708,7 +708,9 @@ class DMSBaseMixin(ACIDMixin):
         return instrument
 
     def _get_opt_element(self):
-        """Get string representation of the optical elements
+        """Get string representation of the optical elements.
+        This includes only elements contained in the filter/pupil
+        wheels of the instrument.
 
         Returns
         -------
@@ -718,7 +720,7 @@ class DMSBaseMixin(ACIDMixin):
         """
         # Retrieve all the optical elements
         opt_elems = []
-        for opt_elem in ['opt_elem', 'opt_elem2', 'opt_elem3']:
+        for opt_elem in ['opt_elem', 'opt_elem2']:
             try:
                 values = list(self.constraints[opt_elem].found_values)
             except KeyError:
@@ -738,6 +740,38 @@ class DMSBaseMixin(ACIDMixin):
 
         return opt_elem
 
+    def _get_slit_name(self):
+        """Get string representation of the slit name (NIRSpec fixed-slit only)
+
+        Returns
+        -------
+        slit_name : str
+            The Level3 Product name representation
+            of the slit name.
+        """
+        # Retrieve all the slit names (sometimes there can be 2)
+        slit_names = []
+        for fxd_slit in ['fxd_slit', 'fxd_slit2']:
+            try:
+                values = list(self.constraints[fxd_slit].found_values)
+            except KeyError:
+                pass
+            else:
+                values.sort(key=str.lower)
+                value = format_list(values)
+                if value not in _EMPTY:
+                    slit_names.append(value)
+
+        # Build the string. Sort the elements in order to
+        # create data-independent results
+        slit_names.sort(key=str.lower)
+        slit_name = '-'.join(slit_names)
+
+        if slit_name == '':
+            slit_name = None
+
+        return slit_name
+
     def _get_subarray(self):
         """Get string representation of the subarray
 
diff --git a/jwst/associations/lib/rules_level2_base.py b/jwst/associations/lib/rules_level2_base.py
index e14f6e53b..3a148fda6 100644
--- a/jwst/associations/lib/rules_level2_base.py
+++ b/jwst/associations/lib/rules_level2_base.py
@@ -839,7 +839,7 @@ class Constraint_Mode(Constraint):
                 required=False,
             ),
             DMSAttrConstraint(
-                name='opt_elem3',
+                name='fxd_slit',
                 sources=['fxd_slit'],
                 required=False,
             ),
diff --git a/jwst/associations/lib/rules_level3.py b/jwst/associations/lib/rules_level3.py
index f2bd5bc8f..9d06b3200 100644
--- a/jwst/associations/lib/rules_level3.py
+++ b/jwst/associations/lib/rules_level3.py
@@ -8,7 +8,7 @@ from jwst.associations.lib.dms_base import (nrccoron_valid_detector)
 from jwst.associations.lib.process_list import ListCategory
 from jwst.associations.lib.rules_level3_base import *
 from jwst.associations.lib.rules_level3_base import (
-    dms_product_name_sources, dms_product_name_noopt, dms_product_name_coronimage,
+    dms_product_name_sources, dms_product_name_nrsfs_sources, dms_product_name_noopt, dms_product_name_coronimage,
     format_product
 )
 
@@ -539,7 +539,7 @@ class Asn_Lv3NRSFSS(AsnMixin_Spectrum):
 
     @property
     def dms_product_name(self):
-        return dms_product_name_sources(self)
+        return dms_product_name_nrsfs_sources(self)
 
 
 @RegistryMarker.rule
diff --git a/jwst/associations/lib/rules_level3_base.py b/jwst/associations/lib/rules_level3_base.py
index a83608949..cd7d8c4ba 100644
--- a/jwst/associations/lib/rules_level3_base.py
+++ b/jwst/associations/lib/rules_level3_base.py
@@ -584,8 +584,8 @@ class Utility():
 format_product = FormatTemplate(
     key_formats={
         'source_id': ['s{:05d}', 's{:s}'],
-        'expspcin': ['{:0>2s}']
-
+        'expspcin': ['{:0>2s}'],
+        'slit_name': ['{:s}']
     }
 )
 
@@ -657,6 +657,51 @@ def dms_product_name_sources(asn):
     return product_name.lower()
 
 
+def dms_product_name_nrsfs_sources(asn):
+    """Produce source-based product names for
+       NIRSpec fixed-slit observations.
+
+    Parameters
+    ---------
+    asn : Association
+        The association for which the product
+        name is to be created.
+
+    Returns
+    -------
+    product_name : str
+        The product name
+    """
+    instrument = asn._get_instrument()
+
+    opt_elem = asn._get_opt_element()
+
+    slit_name = asn._get_slit_name()
+    if slit_name:
+        slit_name = '-' + slit_name
+
+    subarray = asn._get_subarray()
+    if subarray:
+        subarray = '-' + subarray
+
+    product_name_format = (
+        'jw{program}-{acid}'
+        '_{source_id}'
+        '_{instrument}'
+        '_{opt_elem}-{slit_name}{subarray}'
+    )
+    product_name = format_product(
+        product_name_format,
+        program=asn.data['program'],
+        acid=asn.acid.id,
+        instrument=instrument,
+        opt_elem=opt_elem,
+        subarray=subarray,
+    )
+
+    return product_name.lower()
+
+
 def dms_product_name_coronimage(asn):
     """Produce image-based product name
        for coronagraphic data
@@ -808,10 +853,22 @@ class Constraint_Optical_Path(Constraint):
                 sources=['pupil', 'grating'],
                 required=False,
             ),
-            DMSAttrConstraint(
-                name='opt_elem3',
-                sources=['fxd_slit'],
-                required=False,
+            Constraint(
+                [
+                    DMSAttrConstraint(
+                        name='fxd_slit2',
+                        sources=['fxd_slit'],
+                        value=['s200a1|s200a2'],
+                        force_unique=False,
+                        required=False,
+                    ),
+                    DMSAttrConstraint(
+                        name='fxd_slit',
+                        sources=['fxd_slit'],
+                        required=False,
+                    ),
+                ],
+                reduce=Constraint.any
             ),
             DMSAttrConstraint(
                 name='subarray',
diff --git a/jwst/pipeline/calwebb_spec3.py b/jwst/pipeline/calwebb_spec3.py
index 88cbb6321..0899796ed 100644
--- a/jwst/pipeline/calwebb_spec3.py
+++ b/jwst/pipeline/calwebb_spec3.py
@@ -202,13 +202,18 @@ class Spec3Pipeline(Pipeline):
         # Process each source
         for source in sources:
 
-            # If each source is a SourceModelContainer
-            # the output name needs to be updated with the source name.
+            # If each source is a SourceModelContainer,
+            # the output name needs to be updated with the source ID, and potentially
+            # also the slit name (for NIRSpec fixed-slit only).
             if isinstance(source, tuple):
                 source_id, result = source
-                self.output_file = format_product(
-                    output_file, source_id=source_id.lower()
-                )
+                if result[0].meta.exposure.type == "NRS_FIXEDSLIT":
+                    slit_name = self._create_nrsfs_slit_name(result)
+                    self.output_file = format_product(
+                        output_file, source_id=source_id.lower(), slit_name=slit_name)
+                else:
+                    self.output_file = format_product(
+                        output_file, source_id=source_id.lower())
             else:
                 result = source
 
@@ -282,3 +287,21 @@ class Spec3Pipeline(Pipeline):
 
         self.log.info('Ending calwebb_spec3')
         return
+
+
+    def _create_nrsfs_slit_name(self, source_models):
+        """Create the complete slit_name product field for NIRSpec fixed-slit products
+
+        Each unique value of slit name within the list of input source models
+        is appended to the final slit name string.
+        """
+                    
+        slit_names = []
+        slit_names.append(source_models[0].name.lower())
+        for i in range(len(source_models)):
+            name = source_models[i].name.lower()
+            if name not in slit_names:
+                slit_names.append(name)
+        slit_name = "-".join(slit_names)  # append slit names using a dash separator
+
+        return slit_name


Test patch:
diff --git a/jwst/assign_wcs/tests/test_nirspec.py b/jwst/assign_wcs/tests/test_nirspec.py
index e9a34badf..03ee98a3d 100644
--- a/jwst/assign_wcs/tests/test_nirspec.py
+++ b/jwst/assign_wcs/tests/test_nirspec.py
@@ -142,6 +142,7 @@ def create_nirspec_fs_file(grating, filter, lamp="N/A"):
     image[0].header['GWA_XTIL'] = 0.3316612243652344
     image[0].header['GWA_YTIL'] = 0.1260581910610199
     image[0].header['SUBARRAY'] = "FULL"
+    image[0].header['FXD_SLIT'] = "S200A1"
     return image
 
 
diff --git a/jwst/associations/tests/test_level3_spectrographic.py b/jwst/associations/tests/test_level3_spectrographic.py
index 255b6536f..464f52522 100644
--- a/jwst/associations/tests/test_level3_spectrographic.py
+++ b/jwst/associations/tests/test_level3_spectrographic.py
@@ -56,7 +56,7 @@ def nirspec_params_id(fixture_value):
             'o001',
             'spec3',
             r'jw99009-o001_spec3_\d{5}_asn',
-            'jw99009-o001_{source_id}_nirspec_f100lp-g140m-s200a2-s200a2',
+            'jw99009-o001_{source_id}_nirspec_f100lp-g140m-{slit_name}-s200a2',
             set(('science', 'target_acquisition', 'autowave'))
         ),
         (
diff --git a/jwst/resample/tests/test_resample_step.py b/jwst/resample/tests/test_resample_step.py
index a9b2b3cf4..7e2db26f2 100644
--- a/jwst/resample/tests/test_resample_step.py
+++ b/jwst/resample/tests/test_resample_step.py
@@ -38,7 +38,8 @@ def nirspec_rate():
         'name': 'NIRSPEC',
         'gwa_tilt': 37.0610,
         'gwa_xtilt': 0.0001,
-        'gwa_ytilt': 0.0001}
+        'gwa_ytilt': 0.0001,
+        'fixed_slit': 'S200A1'}
     im.meta.subarray = {
         'fastaxis': 1,
         'name': 'SUBS200A1',