Repository: spacetelescope/jwst

Issue:
Pathloss corrections for long MOS slitlets
_Issue [JP-3357](https://jira.stsci.edu/browse/JP-3357) was created on JIRA by [Melanie Clarke](https://jira.stsci.edu/secure/ViewProfile.jspa?name=mclarke):_

This ticket is associated with a known issue in NIRSpec MOS data products: spectra extracted from slitlets consisting of more than 3 shutters exhibit unexpected wavelength-dependent flux discrepancies.

The issue appears to be that the pathloss correction file contains data with explicit corrections for 1 shutter slitlets and 3 shutter slitlets.  If the input data matches one of these two configurations, the correction is applied.  If not, no correction is applied and the pipeline issues a warning like: "WARNING: Cannot find matching pathloss model for slit with5 shutters".

To properly correct arbitrary MSA configurations, the pipeline needs to implement a strategy to compose a correction from the existing reference data.    In particular, the correction derived for a 3-shutter slitlet should be applicable to any slitlet > 1 shutter in length because it considers light from a source that may extend only into adjacent shutters. So, for a source in any of the middle 3 shutters of a 5-shutter slitlet, for example, the pipeline should apply a correction based on the middle shutter of the 3-shutter case in the reference file. If the source is in the top or bottom shutter, then it should use the corresponding top/bottom shutter in the reference file.

Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/spacetelescope/jwst/pull/8106?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) Report
Attention: `55 lines` in your changes are missing coverage. Please review.
> Comparison is base [(`870f9de`)](https://app.codecov.io/gh/spacetelescope/jwst/commit/870f9de3714a7ca111a6b3e5e2465792778a57bb?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) 75.94% compared to head [(`65e6ed9`)](https://app.codecov.io/gh/spacetelescope/jwst/pull/8106?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) 75.83%.

| [Files](https://app.codecov.io/gh/spacetelescope/jwst/pull/8106?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) | Patch % | Lines |
|---|---|---|
| [jwst/pathloss/pathloss.py](https://app.codecov.io/gh/spacetelescope/jwst/pull/8106?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope#diff-andzdC9wYXRobG9zcy9wYXRobG9zcy5weQ==) | 21.42% | [55 Missing :warning: ](https://app.codecov.io/gh/spacetelescope/jwst/pull/8106?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #8106      +/-   ##
==========================================
- Coverage   75.94%   75.83%   -0.11%     
==========================================
  Files         460      460              
  Lines       37625    37691      +66     
==========================================
+ Hits        28573    28584      +11     
- Misses       9052     9107      +55     
```

| [Flag](https://app.codecov.io/gh/spacetelescope/jwst/pull/8106/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [nightly](https://app.codecov.io/gh/spacetelescope/jwst/pull/8106/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) | `77.37% <ø> (-0.01%)` | :arrow_down: | Carriedforward from [870f9de](https://app.codecov.io/gh/spacetelescope/jwst/commit/870f9de3714a7ca111a6b3e5e2465792778a57bb?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) |

*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) to find out more.


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/spacetelescope/jwst/pull/8106?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope).

hbushouse: @stscirij I can't get the regtests to run on this PR branch. Some kind of problem with the branch being based on a very old tag from master. Need to update the PR branch to include all tags. And then please start a regtest run over at https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/
stscirij: Ran a regtest run at
[https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1081/](https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1081/)
nden: > I deleted the tag from my fork to get it to run

What you should do to avoid problems in the future is push all tags to your fork
```
git fetch upstream
git push origin --tags
```

Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index 0bbaa4807..93e775416 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -52,14 +52,6 @@ general
 
 - Moved build configuration from ``setup.cfg`` to ``pyproject.toml`` to support PEP621 [#6847]
 
-outlier_detection
------------------
-
-- Remove use of ``scipy.signal.medfilt`` which is undefined for ``nan``
-  inputs. [#8033]
-
-- Replace uses of ``datetime.utcnow`` (deprecated in python 3.12) [#8051]
-
 imprint
 -------
 
@@ -72,6 +64,19 @@ nsclean
 - Implemented this new step, which is used to remove 1/f noise from NIRSpec
   images. [#8000]
 
+outlier_detection
+-----------------
+
+- Remove use of ``scipy.signal.medfilt`` which is undefined for ``nan``
+  inputs. [#8033]
+
+- Replace uses of ``datetime.utcnow`` (deprecated in python 3.12) [#8051]
+
+pathloss
+--------
+
+- Updated code to handle NIRSpec MOS slitlets that aren't 1X1 or 1X3. [#8106]
+
 photom
 ------
 
diff --git a/docs/jwst/pathloss/description.rst b/docs/jwst/pathloss/description.rst
index e15673475..439d07fd2 100644
--- a/docs/jwst/pathloss/description.rst
+++ b/docs/jwst/pathloss/description.rst
@@ -49,6 +49,63 @@ The form of the 2-D correction (point or uniform) that's appropriate for the
 data is divided into the SCI and ERR arrays and propagated into the variance
 arrays of the science data.
 
+The MSA reference file contains 2 entries: one for a 1x1 slit and one for a 1x3 slit.
+Each entry contains the pathloss correction for point source and uniform sources.
+The former depends on the position of the target in the fiducial shutter and
+wavelength, whereas the latter depends on wavelength only.  The point source 
+entry consists of a 3-d array, where 2 of the dimensions map to the location
+of the source (ranging from -0.5 to 0.5 in both X and Y), while the third dimension
+carries the wavelength dependence.  The 1x3 shutter is 3 times as large in Y as in X.
+
+The entry to use for a point source target is determined by looking at the shutter_state
+attribute of the slit used.  This is a string with a length equal to the number
+of shutters that make up the slit, with 1 denoting an open shutter, 0 a closed
+shutter and x the fiducial (target) shutter.  The reference entry is determined
+by how many shutters next to the fiducial shutter are open:
+
+If both adjacent shutters are closed, the 1x1 entry is used.  A matching
+shutter_state might be 'x' or '10x01'
+
+If both adjacent shutters are open, the center region of the 1x3 entry is used.
+This would be the case for a slit with shutter state '1x1' or '1011x1'.
+
+If one adjacent shutter is open and one closed, the 1x3 entry is used.  If the
+shutter below the fiducial is open and the shutter above closed, then the upper
+region of the 1x3 pathloss array is used.  This is implemented by adding 1 to the
+Y coordinate of the target position (bringing it into the range +0.5 to +1.5),
+moving it to the upper third of the pathloss array.  A matching shutter state
+might be '1x' or '11x011'
+
+Similarly, if the shutter below the fiducial is closed and that above is open, the
+lower third of the pathloss array is used by subtracting 1 from the Y coordinate of
+the target position (bringing it into the range -1.5 to -0.5).  A matching shutter
+state could be 'x111' or '110x1'.
+
+Once the X and Y coordinates of the source are mapped into a pixel location in the
+spatial dimensions of the pathloss array using the WCS of the transformation of position
+to pixel location, the wavelength dependence is determined
+by interpolating at that (fractional) pixel position in each wavelength plane,
+resulting in a pair of 1-d arrays of pathloss correction and wavelength.  These arrays
+are used to interpolate the correction for each pixel of the 2-d extracted science
+array, since each pixel has a different wavelength, and the correction is applied
+to the science pixel array.
+
+For uniform sources, there is no dependence of the pathloss correction on position,
+so the correction arrays are just 1-d arrays of correction and wavelength.  Once
+again, the shutter_state attribute of each slit is used to determine the correction
+entry used:
+
+If both shutters adjacent to the fiducial are closed, the 1x1 entry is used
+
+If both shutters adjacent to the fiducial are open, the 1x3 entry is used
+
+If one is closed and one is open, the correction used is the average of the 1x1
+and 1x3 entries.
+
+Like for the point source case, the 1-d arrays of pathloss correction and wavelength
+are used to interpolate the correction for each pixel in the science data, using the
+wavelength of each pixel to interpolate into the pathloss correction array.
+
 MIRI LRS
 ++++++++
 The algorithm for MIRI LRS mode is largely the same as that for NIRSpec described
diff --git a/jwst/pathloss/pathloss.py b/jwst/pathloss/pathloss.py
index 01ea719ba..6b52e83c7 100644
--- a/jwst/pathloss/pathloss.py
+++ b/jwst/pathloss/pathloss.py
@@ -98,9 +98,26 @@ def get_center(exp_type, input, offsets=False):
         return 0.0, 0.0
 
 
+def shutter_above_is_closed(shutter_state):
+    ref_loc = shutter_state.find('x')
+    nshutters = len(shutter_state)
+    if ref_loc == nshutters - 1 or shutter_state[ref_loc + 1] == '0':
+        return True
+    else:
+        return False
+
+
+def shutter_below_is_closed(shutter_state):
+    ref_loc = shutter_state.find('x')
+    if ref_loc == 0 or shutter_state[ref_loc - 1] == '0':
+        return True
+    else:
+        return False
+
+
 def get_aperture_from_model(input_model, match):
     """Figure out the correct aperture based on the value of the 'match'
-    parameter.  For MSA, match is the number of shutters, for fixed slit,
+    parameter.  For MSA, match is the shutter state string, for fixed slit,
     match is the name of the slit.
 
     Parameters
@@ -109,7 +126,7 @@ def get_aperture_from_model(input_model, match):
         science data to be corrected
 
     match : str
-        Aperture name
+        Aperture name or shutter state
 
     Returns
     -------
@@ -117,8 +134,15 @@ def get_aperture_from_model(input_model, match):
         Aperture name
     """
     if input_model.meta.exposure.type == 'NRS_MSASPEC':
+        # Currently there are only 2 apertures in the MSA pathloss reference file: 1x1 and 1x3
+        # Only return the 1x1 aperture if the reference shutter has closed shutters above and below
+        if shutter_below_is_closed(match) and shutter_above_is_closed(match):
+            matchsize = 1
+        else:
+            matchsize = 3
         for aperture in input_model.apertures:
-            if aperture.shutters == match:
+            # Only return the aperture
+            if aperture.shutters == matchsize:
                 return aperture
     elif input_model.meta.exposure.type in ['NRS_FIXEDSLIT', 'NRS_BRIGHTOBJ',
                                             'NIS_SOSS']:
@@ -237,6 +261,60 @@ def calculate_pathloss_vector(pathloss_refdata,
         return wavelength, pathloss_vector, is_inside_slitlet
 
 
+def calculate_two_shutter_uniform_pathloss(pathloss_model):
+    """The two shutter MOS case for uniform source calculation requires a custom
+     routine since it uses both the 1X1 and 1X3 extensions of the pathloss reference file
+
+    Parameters
+    ----------
+    pathloss_model : pathloss datamodel
+        The pathloss datamodel
+
+    Returns
+    -------
+    (wavelength, pathloss_vector) : tuple of 2 1-d numpy arrays
+        The wavelength and pathloss 1-d arrays
+
+    """
+    # This routine will run if the fiducial shutter has 1 adjacent open shutter
+    n_apertures = len(pathloss_model.apertures)
+    if n_apertures != 2:
+        log.warning(f"Expected 2 apertures in pathloss reference file, found {n_apertures}")
+        return (None, None)
+    for aperture in pathloss_model.apertures:
+        aperture_name = aperture.name.upper()
+        if aperture_name == 'MOS1X1':
+            aperture1x1 = aperture
+        elif aperture_name == 'MOS1X3':
+            aperture1x3 = aperture
+        if aperture_name not in ['MOS1X1', 'MOS1X3']:
+            log.warning(f"Unexpected aperture name {aperture_name} (Expected 'MOS1X1' or 'MOS1X3')")
+            return (None, None)
+    pathloss1x1 = aperture1x1.uniform_data
+    pathloss1x3 = aperture1x3.uniform_data
+    if len(pathloss1x1) != len(pathloss1x3):
+        log.warning("Pathloss 1x1 and 1x3 arrays have different sizes")
+        return (None, None)
+    if aperture1x1.uniform_wcs.crval1 != aperture1x3.uniform_wcs.crval1:
+        log.warning("1x1 and 1x3 apertures have different WCS CRVAL1")
+        return (None, None)
+    if aperture1x1.uniform_wcs.crpix1 != aperture1x3.uniform_wcs.crpix1:
+        log.warning("1x1 and 1x3 apertures have different WCS CRPIX1")
+        return (None, None)
+    if aperture1x1.uniform_wcs.cdelt1 != aperture1x3.uniform_wcs.cdelt1:
+        log.warning("1x1 and 1x3 apertures have different WCS CDELT1")
+        return (None, None)
+    wavesize = len(pathloss1x1)
+    wavelength = np.zeros(wavesize)
+    crpix1 = aperture1x1.uniform_wcs.crpix1
+    crval1 = aperture1x1.uniform_wcs.crval1
+    cdelt1 = aperture1x1.uniform_wcs.cdelt1
+    for i in np.arange(wavesize):
+        wavelength[i] = crval1 + (float(i + 1) - crpix1) * cdelt1
+    average_pathloss = 0.5 * (pathloss1x1 + pathloss1x3)
+    return (wavelength, average_pathloss)
+
+
 def do_correction(input_model, pathloss_model=None, inverse=False, source_type=None,
                   correction_pars=None, user_slit_loc=None):
     """Execute all tasks for Path Loss Correction
@@ -752,19 +830,41 @@ def _corrections_for_mos(slit, pathloss, exp_type, source_type=None):
         # Calculate the 1-d wavelength and pathloss vectors
         # for the source position
         # Get the aperture from the reference file that matches the slit
-        nshutters = util.get_num_msa_open_shutters(slit.shutter_state)
-        aperture = get_aperture_from_model(pathloss, nshutters)
+        slitlength = len(slit.shutter_state)
+        aperture = get_aperture_from_model(pathloss, slit.shutter_state)
+        log.info(f"Shutter state = {slit.shutter_state}, using {aperture.name} entry in ref file")
+        two_shutters = False
+        if shutter_below_is_closed(slit.shutter_state) and not shutter_above_is_closed(slit.shutter_state):
+            ycenter = ycenter - 1.0
+            log.info('Shutter below fiducial is closed, using lower region of pathloss array')
+            two_shutters = True
+        if not shutter_below_is_closed(slit.shutter_state) and shutter_above_is_closed(slit.shutter_state):
+            ycenter = ycenter + 1.0
+            log.info('Shutter above fiducial is closed, using upper region of pathloss array')
+            two_shutters = True
         if aperture is not None:
             (wavelength_pointsource,
              pathloss_pointsource_vector,
              is_inside_slitlet) = calculate_pathloss_vector(aperture.pointsource_data,
                                                             aperture.pointsource_wcs,
                                                             xcenter, ycenter)
-            (wavelength_uniformsource,
-             pathloss_uniform_vector,
-             dummy) = calculate_pathloss_vector(aperture.uniform_data,
-                                                aperture.uniform_wcs,
-                                                xcenter, ycenter)
+            if two_shutters:
+                (wavelength_uniformsource,
+                 pathloss_uniform_vector) = calculate_two_shutter_uniform_pathloss(pathloss)
+            else:
+                (wavelength_uniformsource,
+                 pathloss_uniform_vector,
+                 dummy) = calculate_pathloss_vector(aperture.uniform_data,
+                                                    aperture.uniform_wcs,
+                                                    xcenter, ycenter)
+            # This should only happen if the 2 shutter uniform pathloss calculation has an error
+            if wavelength_uniformsource is None or pathloss_uniform_vector is None:
+                log.warning("Unable to calculate 2 shutter uniform pathloss, using 3 shutter aperture")
+                (wavelength_uniformsource,
+                 pathloss_uniform_vector,
+                 dummy) = calculate_pathloss_vector(aperture.uniform_data,
+                                                    aperture.uniform_wcs,
+                                                    xcenter, ycenter)
             if is_inside_slitlet:
 
                 # Wavelengths in the reference file are in meters,
@@ -801,7 +901,7 @@ def _corrections_for_mos(slit, pathloss, exp_type, source_type=None):
                 log.warning("Source is outside slit.")
         else:
             log.warning("Cannot find matching pathloss model for slit with"
-                        f"{nshutters} shutters")
+                        f"{slitlength} shutters")
     else:
         log.warning(f"Slit has data size = {size}")
 


Test patch:
diff --git a/jwst/pathloss/tests/test_pathloss.py b/jwst/pathloss/tests/test_pathloss.py
index c45fd01f2..b32d6ffba 100644
--- a/jwst/pathloss/tests/test_pathloss.py
+++ b/jwst/pathloss/tests/test_pathloss.py
@@ -8,7 +8,9 @@ from jwst.pathloss.pathloss import (calculate_pathloss_vector,
                                     get_aperture_from_model,
                                     get_center,
                                     interpolate_onto_grid,
-                                    is_pointsource)
+                                    is_pointsource,
+                                    shutter_below_is_closed,
+                                    shutter_above_is_closed)
 from jwst.pathloss.pathloss import do_correction
 import numpy as np
 
@@ -84,10 +86,10 @@ def test_get_aper_from_model_msa():
     aperture reference data is returned for MSA mode"""
 
     datmod = PathlossModel()
-    datmod.apertures.append({'shutters': 5})
+    datmod.apertures.append({'shutters': 3})
     datmod.meta.exposure.type = 'NRS_MSASPEC'
 
-    result = get_aperture_from_model(datmod, 5)
+    result = get_aperture_from_model(datmod, '11x11')
 
     assert result == datmod.apertures[0]
 
@@ -335,3 +337,21 @@ def test_interpolate_onto_grid():
     result_comparison = np.interp(wavelength_grid, extended_wavelength_vector, extended_pathloss_vector)
 
     np.testing.assert_array_equal(result, result_comparison)
+
+
+def test_shutter_below_is_closed():
+    shutter_below_closed = ['x111', 'x', '10x11']
+    shutter_below_open = ['11x11', '111x', '11x01']
+    for shutter_state in shutter_below_closed:
+        assert shutter_below_is_closed(shutter_state)
+    for shutter_state in shutter_below_open:
+        assert not shutter_below_is_closed(shutter_state)
+
+
+def test_shutter_above_is_closed():
+    shutter_above_closed = ['111x', 'x', '1x011']
+    shutter_above_open = ['11x11', 'x111', '110x1']
+    for shutter_state in shutter_above_closed:
+        assert shutter_above_is_closed(shutter_state)
+    for shutter_state in shutter_above_open:
+        assert not shutter_above_is_closed(shutter_state)