Repository: spacetelescope/jwst

Issue:
Review jwst code for step.skip usage
_Issue [JP-3631](https://jira.stsci.edu/browse/JP-3631) was created on JIRA by [Melanie Clarke](https://jira.stsci.edu/secure/ViewProfile.jspa?name=mclarke):_

Steps defined in stpipe always have a 'skip' attribute; when this is set to True, the step is skipped.  This is available to users as a top-level parameter for each step.

The skip attribute should rarely be set by code within a pipeline step, since it may have unintended consequences.  For example, currently, when the outlier_detection step is run on NIRSpec MOS data, if skip is set for one slit, the step is then skipped for all subsequent slits.  This fix is addressed in the PR for JP-2922.  A similar fix for the tweakreg step is addressed in PR 8476 ([https://github.com/spacetelescope/jwst/pull/8476).](https://github.com/spacetelescope/jwst/pull/8476)

We should review remaining steps for similar issues and make sure that the skip attribute is only set by the user or at the pipeline level, not internal to any step.  In particular, it looks like skip is set to True in the core class JwstStep when record_step_status is called with success=False.  This function should probably either be deprecated or updated to avoid setting skip directly.

Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/spacetelescope/jwst/pull/8600?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) Report
Attention: Patch coverage is `27.27273%` with `8 lines` in your changes missing coverage. Please review.
> Project coverage is 59.56%. Comparing base [(`23fb5cd`)](https://app.codecov.io/gh/spacetelescope/jwst/commit/23fb5cd77e2f9a0d0ca34c9027ce6c6979ae45e7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) to head [(`5856a4b`)](https://app.codecov.io/gh/spacetelescope/jwst/commit/5856a4b7d6d9b8c22bf7df6109915dece4cfabcf?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope).
> Report is 1 commits behind head on master.

| [Files](https://app.codecov.io/gh/spacetelescope/jwst/pull/8600?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) | Patch % | Lines |
|---|---|---|
| [jwst/tweakreg/tweakreg\_step.py](https://app.codecov.io/gh/spacetelescope/jwst/pull/8600?src=pr&el=tree&filepath=jwst%2Ftweakreg%2Ftweakreg_step.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope#diff-andzdC90d2Vha3JlZy90d2Vha3JlZ19zdGVwLnB5) | 0.00% | [6 Missing :warning: ](https://app.codecov.io/gh/spacetelescope/jwst/pull/8600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) |
| [jwst/cube\_build/cube\_build\_step.py](https://app.codecov.io/gh/spacetelescope/jwst/pull/8600?src=pr&el=tree&filepath=jwst%2Fcube_build%2Fcube_build_step.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope#diff-andzdC9jdWJlX2J1aWxkL2N1YmVfYnVpbGRfc3RlcC5weQ==) | 50.00% | [1 Missing :warning: ](https://app.codecov.io/gh/spacetelescope/jwst/pull/8600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) |
| [jwst/pipeline/calwebb\_spec3.py](https://app.codecov.io/gh/spacetelescope/jwst/pull/8600?src=pr&el=tree&filepath=jwst%2Fpipeline%2Fcalwebb_spec3.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope#diff-andzdC9waXBlbGluZS9jYWx3ZWJiX3NwZWMzLnB5) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/spacetelescope/jwst/pull/8600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #8600      +/-   ##
==========================================
+ Coverage   59.54%   59.56%   +0.02%     
==========================================
  Files         391      391              
  Lines       39292    39278      -14     
==========================================
+ Hits        23396    23397       +1     
+ Misses      15896    15881      -15     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/spacetelescope/jwst/pull/8600?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope).

emolter: new regtest round [started here](https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1566/)
melanieclarke: > In terms of how to fix that, should I allow `nirspec_utils.is_background_msa_slit()` to check for "BACKGROUND" or "BKG", like I'm doing here in the most recent commit, or should we change the test dataset?

I think it wouldn't hurt to allow either, in case there are other older datasets around.
braingram: Did you notice if any steps are now saving results when skipped (when they weren't previously)? I ask because stpipe checks `Step.skip` when saving resulting:
https://github.com/spacetelescope/stpipe/blob/915a8326432254bd5d03ee67586bc94c79d74327/src/stpipe/step.py#L536
I would expect that, for example, tweakreg would now save results if provided an input association with 1 group when configured to skip absolute alignment (to trigger the skip on [line 208](https://github.com/spacetelescope/jwst/pull/8600/files#diff-4437d46a04f3e795cee0295ff4b11d91f8c618cbe0e3c9fa7a3751b74e575db1R208)).
emolter: Putting some relevant points from a meeting Brett and I had here:

[This line](https://github.com/spacetelescope/stpipe/blob/01f1956f3b3c3750a26790d5217cf909cad4b95a/src/stpipe/step.py#L536) in stpipe is checking the `step.skip` parameter to decide whether files should be saved when `save_results` is True.  With the changes in this PR, that means that if `save_results==True` but a step status is set to SKIPPED, output files will be created when they would not have before.  We agreed that this should be handled in `stpipe` somehow, perhaps by making `step.finalize_result()` check the metadata of the result and returning a value that indicates whether the step had been skipped.

We discussed whether `record_step_status` and `query_step_status` should be put into stpipe, along with variables similar to the one suggested in [this comment](https://github.com/spacetelescope/jwst/pull/8600#discussion_r1657223680) for the SKIPPED, COMPLETE, and NOTSET constants.  In a perfect world it would be good to achieve similarity between Roman and JWST in the way their pipeline steps are set up and the way stpipe handles them, e.g. [this open issue](https://github.com/spacetelescope/roman_datamodels/issues/343) which is a problem for Roman but not Webb.  What do other people think about moving these to stpipe?

If we do decide to do something in stpipe, it remains unclear whether this should hold up merging this PR.  If the creation of files when `save_results` is True even when steps are skipped is not a serious problem, then I'd say we can go ahead and move toward merging it.  If it is a serious problem, then we should coordinate this PR with an stpipe release that includes at minimum the aforementioned change to `finalize_result`.  What do you think @nden @hbushouse?
melanieclarke: > If the creation of files when `save_results` is True even when steps are skipped is not a serious problem, then I'd say we can go ahead and move toward merging it.

This actually sounds like the more desirable behavior to me -- if save_results is True, then it was either True by default, in which case it is a required output file, or else it was True because the user explicitly set it and is expecting an output file to be saved.  Even if it's only different from the input file by a header keyword, it's sometimes useful to save an output file at an intermediate point in the pipeline, whether or not the step immediately prior actually did anything useful.

In either case, I don't think it's a serious enough problem to hold up the resolution for this ticket.
emolter: The style check failure seems to be due to some kind of version bump of `ruff`; does anyone know anything about that?

```
ruff==0.5.0
check-style: commands[0]> ruff .
error: `ruff <path>` has been removed. Use `ruff check <path>` instead.
```
braingram: > The style check failure seems to be due to some kind of version bump of `ruff`; does anyone know anything about that?
> 
> ```
> ruff==0.5.0
> check-style: commands[0]> ruff .
> error: `ruff <path>` has been removed. Use `ruff check <path>` instead.
> ```

Wee woo wee woo, calling @zacharyburnett :) Have you seen this anywhere else? Thanks!
zacharyburnett: > > The style check failure seems to be due to some kind of version bump of `ruff`; does anyone know anything about that?
> > ```
> > ruff==0.5.0
> > check-style: commands[0]> ruff .
> > error: `ruff <path>` has been removed. Use `ruff check <path>` instead.
> > ```
> 
> Wee woo wee woo, calling @zacharyburnett :) Have you seen this anywhere else? Thanks!

oh yes! `ruff` uses `ruff check <path>` now (because they added `ruff format <path>` as well); I've updated several other style checks in other projects to use that, but I must have missed this one
emolter: [regression tests again](https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1570/)
emolter: [even more regtests](https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1572/). I will merge this if these don't turn up any failures
emolter: This is ready to merge.  Waiting because this should go into Build 11.1, and other bugfixes still need to go into 11.0

Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index c18dce5a1..72b9f369e 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -1,7 +1,40 @@
-1.15.1 (unreleased)
+1.15.2 (unreleased)
 ===================
 
--
+cube_build
+----------
+
+- Removed direct setting of the ``self.skip`` attribute from within the step
+  itself. [#8600]
+
+master_background
+-----------------
+
+- Either of ``"background"`` or ``"bkg"`` in slit name now defines the slit
+  as a background slit, instead of ``"bkg"`` only. [#8600]
+
+stpipe
+------
+
+- Removed setting of the `self.skip` attribute in the `record_step_status()` function;
+  added a `query_step_status()` function to use as an alternative to checking 
+  `self.skip`. [#8600]
+
+tweakreg
+--------
+
+- Removed direct setting of the ``self.skip`` attribute from within the step
+  itself. [#8600]
+
+1.15.1 (2024-07-08)
+===================
+
+ramp_fitting
+------------
+
+- Fix bugs in the C algorithm Poisson variance calculation when provided with an average dark current. [stcal#269]
+
+- Use the C extension with multiprocessing. [stcal#268]
 
 1.15.0 (2024-06-26)
 ===================
diff --git a/CITATION.cff b/CITATION.cff
index 7f4b7b3a4..f34b4c8c9 100644
--- a/CITATION.cff
+++ b/CITATION.cff
@@ -74,8 +74,11 @@ authors:
 - family-names: "Pena-Guerrero"
   given-names: "Maria"
   orcid: "https://orcid.org/0000-0003-2314-3453"
+- family-names: "Graham"
+  given-names: "Brett"
+  orcid: "https://orcid.org/0000-0001-6315-4507"
 title: "JWST Calibration Pipeline"
-version: 1.15.0
+version: 1.15.1
 doi: 10.5281/zenodo.7038885
-date-released: 2024-06-26
+date-released: 2024-07-08
 url: "https://github.com/spacetelescope/jwst"
diff --git a/README.md b/README.md
index b57f62af8..a83e87bd3 100644
--- a/README.md
+++ b/README.md
@@ -216,7 +216,9 @@ the specified context and less than the context for the next release.
 
 | jwst tag            | DMS build | SDP_VER  | CRDS_CONTEXT | Released   | Ops Install | Notes                                         |
 |---------------------|-----------|----------|--------------|------------|-------------|-----------------------------------------------|
+| 1.15.1              | B11.0rc2  | TBD      | 1242         | 2024-07-08 | TBD         | Second release candidate for B11.0            |
 | 1.15.0              | B11.0rc1  | TBD      | 1241         | 2024-06-26 | TBD         | First release candidate for B11.0             |
+| 1.14.1              |           |          | 1238         | 2024-06-27 |             | PyPI-only release for external users          |
 | 1.14.0              | B10.2.1   | 2024.1.1 | 1238         | 2024-03-29 | 2024-06-12  | Final release candidate for B10.2.1           |
 | 1.13.4              |           |          | 1185         | 2024-01-25 |             | PyPI-only release for external users          |
 | 1.13.3              | B10.1     | 2023.4.0 | 1181         | 2024-01-05 |             | Final release candidate for B10.1             |
diff --git a/jwst/cube_build/cube_build_step.py b/jwst/cube_build/cube_build_step.py
index 15a9dde45..7801169ba 100755
--- a/jwst/cube_build/cube_build_step.py
+++ b/jwst/cube_build/cube_build_step.py
@@ -3,7 +3,7 @@
 
 from jwst.datamodels import ModelContainer
 
-from ..stpipe import Step
+from ..stpipe import Step, record_step_status
 from . import cube_build
 from . import ifu_cube
 from . import data_types
@@ -396,14 +396,13 @@ class CubeBuildStep (Step):
                 if key in cube.meta.wcsinfo.instance:
                     del cube.meta.wcsinfo.instance[key]
         if status_cube == 1:
-            self.skip = True
+            record_step_status(cube_container, "cube_build", success=False)
+        else:
+            record_step_status(cube_container, "cube_build", success=True)
 
         t1 = time.time()
         self.log.debug(f'Time to build all cubes {t1-t0}')
 
-        if status_cube == 1:
-            self.skip = True
-
         input_table.close()
         return cube_container
 # ******************************************************************************
diff --git a/jwst/master_background/master_background_mos_step.py b/jwst/master_background/master_background_mos_step.py
index 252155a3c..45c641ff5 100644
--- a/jwst/master_background/master_background_mos_step.py
+++ b/jwst/master_background/master_background_mos_step.py
@@ -1,5 +1,6 @@
 """Master Background Pipeline for applying Master Background to NIRSpec MOS data"""
 from stpipe.step import preserve_step_pars
+from jwst.stpipe import record_step_status
 
 from stdatamodels.jwst import datamodels
 
@@ -108,7 +109,7 @@ class MasterBackgroundMosStep(Pipeline):
             if not self.force_subtract and \
                'COMPLETE' in [data_model.meta.cal_step.back_sub, data_model.meta.cal_step.master_background]:
                 self.log.info('Background subtraction has already occurred. Skipping.')
-                self.record_step_status(data, 'master_background', False)
+                record_step_status(data, 'master_background', success=False)
                 return data
 
             if self.user_background:
@@ -123,11 +124,11 @@ class MasterBackgroundMosStep(Pipeline):
                 num_bkg, num_src = self._classify_slits(data_model)
                 if num_bkg == 0:
                     self.log.warning('No background slits available for creating master background. Skipping')
-                    self.record_step_status(data, 'master_background', False)
+                    record_step_status(data, 'master_background', False)
                     return data
                 elif num_src == 0:
                     self.log.warning('No source slits for applying master background. Skipping')
-                    self.record_step_status(data, 'master_background', False)
+                    record_step_status(data, 'master_background', False)
                     return data
 
                 self.log.info('Calculating master background')
@@ -136,14 +137,14 @@ class MasterBackgroundMosStep(Pipeline):
             # Check that a master background was actually determined.
             if master_background is None:
                 self.log.info('No master background could be calculated. Skipping.')
-                self.record_step_status(data, 'master_background', False)
+                record_step_status(data, 'master_background', False)
                 return data
 
             # Now apply the de-calibrated background to the original science
             result = nirspec_utils.apply_master_background(data_model, mb_multislit, inverse=self.inverse)
 
             # Mark as completed and setup return data
-            self.record_step_status(result, 'master_background', True)
+            record_step_status(result, 'master_background', True)
             self.correction_pars = {
                 'masterbkg_1d': master_background,
                 'masterbkg_2d': mb_multislit
diff --git a/jwst/master_background/master_background_step.py b/jwst/master_background/master_background_step.py
index 76efbbff6..1ea83855e 100755
--- a/jwst/master_background/master_background_step.py
+++ b/jwst/master_background/master_background_step.py
@@ -4,6 +4,7 @@ from stdatamodels.properties import merge_tree
 from stdatamodels.jwst import datamodels
 
 from jwst.datamodels import ModelContainer
+from jwst.stpipe import record_step_status
 
 from ..stpipe import Step
 from ..combine_1d.combine1d import combine_1d_spectra
@@ -63,7 +64,7 @@ class MasterBackgroundStep(Step):
             # First check if we should even do the subtraction.  If not, bail.
             if not self._do_sub:
                 result = input_data.copy()
-                self.record_step_status(result, 'master_background', success=False)
+                record_step_status(result, 'master_background', success=False)
                 return result
 
             # Check that data is a supported datamodel. If not, bail.
@@ -78,7 +79,7 @@ class MasterBackgroundStep(Step):
                     "Input %s of type %s cannot be handled.  Step skipped.",
                     input, type(input)
                 )
-                self.record_step_status(result, 'master_background', success=False)
+                record_step_status(result, 'master_background', success=False)
                 return result
 
             # If user-supplied master background, subtract it
@@ -159,7 +160,7 @@ class MasterBackgroundStep(Step):
                         "Input %s of type %s cannot be handled without user-supplied background.  Step skipped.",
                         input, type(input)
                     )
-                    self.record_step_status(result, 'master_background', success=False)
+                    record_step_status(result, 'master_background', success=False)
                     return result
 
                 # Save the computed background if requested by user
@@ -167,7 +168,7 @@ class MasterBackgroundStep(Step):
                     self.save_model(master_background, suffix='masterbg1d', force=True, asn_id=asn_id)
                     self.save_model(background_2d_collection, suffix='masterbg2d', force=True, asn_id=asn_id)
 
-            self.record_step_status(result, 'master_background', success=True)
+            record_step_status(result, 'master_background', success=True)
 
         return result
 
diff --git a/jwst/master_background/nirspec_utils.py b/jwst/master_background/nirspec_utils.py
index d29c9da26..153bf9f50 100644
--- a/jwst/master_background/nirspec_utils.py
+++ b/jwst/master_background/nirspec_utils.py
@@ -254,7 +254,8 @@ def is_background_msa_slit(slit):
     bool
         True if the slit is background; False if it is not.
     """
-    if "BKG" in str(slit.source_name).upper():
+    name = str(slit.source_name).upper()
+    if ("BKG" in name) or ("BACKGROUND" in name):
         return True
     else:
         return False
diff --git a/jwst/pipeline/calwebb_spec2.py b/jwst/pipeline/calwebb_spec2.py
index 9f3004f4a..d75f44dc7 100644
--- a/jwst/pipeline/calwebb_spec2.py
+++ b/jwst/pipeline/calwebb_spec2.py
@@ -5,6 +5,7 @@ import traceback
 import numpy as np
 
 from stdatamodels.jwst import datamodels
+from jwst.stpipe import query_step_status
 
 from ..assign_wcs.util import NoDataOnDetectorError
 from ..lib.exposure_types import is_nrs_ifu_flatlamp, is_nrs_ifu_linelamp, is_nrs_slit_linelamp
@@ -332,7 +333,7 @@ class Spec2Pipeline(Pipeline):
             # as DO_NOT_USE or NON_SCIENCE.
             resampled = self.pixel_replace(resampled)
             resampled = self.cube_build(resampled)
-            if not self.cube_build.skip:
+            if query_step_status(resampled, "cube_build") == 'COMPLETE':
                 self.save_model(resampled[0], suffix='s3d')
         elif exp_type in ['MIR_LRS-SLITLESS']:
             resampled = calibrated.copy()
@@ -346,7 +347,7 @@ class Spec2Pipeline(Pipeline):
             # as DO_NOT_USE or NON_SCIENCE.
             resampled = self.pixel_replace(resampled)
         # Extract a 1D spectrum from the 2D/3D data
-        if exp_type in ['MIR_MRS', 'NRS_IFU'] and self.cube_build.skip:
+        if exp_type in ['MIR_MRS', 'NRS_IFU'] and query_step_status(resampled, "cube_build") == 'SKIPPED':
             # Skip extract_1d for IFU modes where no cube was built
             self.extract_1d.skip = True
 
diff --git a/jwst/pipeline/calwebb_spec3.py b/jwst/pipeline/calwebb_spec3.py
index 9357903a1..23b9902a7 100644
--- a/jwst/pipeline/calwebb_spec3.py
+++ b/jwst/pipeline/calwebb_spec3.py
@@ -6,6 +6,7 @@ import os.path as op
 from stdatamodels.jwst import datamodels
 
 from jwst.datamodels import SourceModelContainer
+from jwst.stpipe import query_step_status
 
 from ..associations.lib.rules_level3_base import format_product
 from ..exp_to_source import multislit_to_container
@@ -148,7 +149,7 @@ class Spec3Pipeline(Pipeline):
 
             # If the step is skipped, do the container splitting that
             # would've been done in master_background
-            if self.master_background.skip:
+            if query_step_status(source_models, "master_background") == 'SKIPPED':
                 source_models, bkg_models = split_container(input_models)
                 # we don't need the background members
                 bkg_models.close()
diff --git a/jwst/pixel_replace/pixel_replace_step.py b/jwst/pixel_replace/pixel_replace_step.py
index 3efd7d231..2c9b117e1 100644
--- a/jwst/pixel_replace/pixel_replace_step.py
+++ b/jwst/pixel_replace/pixel_replace_step.py
@@ -2,6 +2,7 @@
 from functools import partial
 
 from ..stpipe import Step
+from jwst.stpipe import record_step_status
 from .. import datamodels
 from .pixel_replace import PixelReplacement
 
@@ -107,7 +108,7 @@ class PixelReplaceStep(Step):
                         replacement = PixelReplacement(model, **pars)
                         replacement.replace()
                         output_model[i] = replacement.output
-                        self.record_step_status(output_model[i], 'pixel_replace', success=True)
+                        record_step_status(output_model[i], 'pixel_replace', success=True)
                 return output_model
             # ________________________________________
             # calewbb_spec2 case - single input model
@@ -117,6 +118,6 @@ class PixelReplaceStep(Step):
                 result = input_model.copy()
                 replacement = PixelReplacement(result, **pars)
                 replacement.replace()
-                self.record_step_status(replacement.output, 'pixel_replace', success=True)
+                record_step_status(replacement.output, 'pixel_replace', success=True)
                 result = replacement.output
                 return result
diff --git a/jwst/stpipe/__init__.py b/jwst/stpipe/__init__.py
index 637a36d0a..8e30727f8 100644
--- a/jwst/stpipe/__init__.py
+++ b/jwst/stpipe/__init__.py
@@ -1,4 +1,5 @@
 from .core import JwstStep as Step, JwstPipeline as Pipeline
+from .utilities import record_step_status, query_step_status
 
 
-__all__ = ['Step', 'Pipeline']
+__all__ = ['Step', 'Pipeline', 'record_step_status', 'query_step_status']
diff --git a/jwst/stpipe/core.py b/jwst/stpipe/core.py
index 01ca6921a..2c86df37c 100644
--- a/jwst/stpipe/core.py
+++ b/jwst/stpipe/core.py
@@ -1,7 +1,6 @@
 """
 JWST-specific Step and Pipeline base classes.
 """
-from collections.abc import Sequence
 from stdatamodels.jwst.datamodels import JwstDataModel
 from stdatamodels.jwst import datamodels
 
@@ -89,33 +88,6 @@ class JwstStep(Step):
                 if self.parent is None:
                     log.info(f"Results used CRDS context: {result.meta.ref_file.crds.context_used}")
 
-    def record_step_status(self, datamodel, cal_step, success=True):
-        """Record whether or not a step completed in meta.cal_step
-
-        Parameters
-        ----------
-        datamodel : `~jwst.datamodels.JwstDataModel` instance
-            This is the datamodel or container of datamodels to modify in place
-
-        cal_step : str
-            The attribute in meta.cal_step for recording the status of the step
-
-        success : bool
-            If True, then 'COMPLETE' is recorded.  If False, then 'SKIPPED'
-        """
-        if success:
-            status = 'COMPLETE'
-        else:
-            status = 'SKIPPED'
-            self.skip = True
-
-        if isinstance(datamodel, Sequence):
-            for model in datamodel:
-                model.meta.cal_step._instance[cal_step] = status
-        else:
-            datamodel.meta.cal_step._instance[cal_step] = status
-
-        # TODO: standardize cal_step naming to point to the official step name
 
     def remove_suffix(self, name):
         return remove_suffix(name)
diff --git a/jwst/stpipe/utilities.py b/jwst/stpipe/utilities.py
index f75af200b..67a476911 100644
--- a/jwst/stpipe/utilities.py
+++ b/jwst/stpipe/utilities.py
@@ -35,6 +35,7 @@ import inspect
 import logging
 import os
 import re
+from collections.abc import Sequence
 
 # Configure logging
 logger = logging.getLogger(__name__)
@@ -51,6 +52,9 @@ NON_STEPS = [
     'SystemCall',
 ]
 
+NOT_SET = "NOT SET"
+COMPLETE = "COMPLETE"
+SKIPPED = "SKIPPED"
 
 def all_steps():
     """List all classes subclassed from Step
@@ -144,3 +148,60 @@ def folder_traverse(folder_path, basename_regex='.+', path_exclude_regex='^$'):
         for file in files:
             if basename_regex.match(file):
                 yield os.path.join(root, file)
+
+
+def record_step_status(datamodel, cal_step, success=True):
+    """Record whether or not a step completed in meta.cal_step
+
+    Parameters
+    ----------
+    datamodel : `~jwst.datamodels.JwstDataModel` instance
+        This is the datamodel or container of datamodels to modify in place
+
+    cal_step : str
+        The attribute in meta.cal_step for recording the status of the step
+
+    success : bool
+        If True, then 'COMPLETE' is recorded.  If False, then 'SKIPPED'
+    """
+    if success:
+        status = COMPLETE
+    else:
+        status = SKIPPED
+
+    if isinstance(datamodel, Sequence):
+        for model in datamodel:
+            model.meta.cal_step._instance[cal_step] = status
+    else:
+        datamodel.meta.cal_step._instance[cal_step] = status
+
+    # TODO: standardize cal_step naming to point to the official step name
+
+
+def query_step_status(datamodel, cal_step):
+    """Query the status of a step in meta.cal_step
+    
+    Parameters
+    ----------
+    datamodel : `~jwst.datamodels.JwstDataModel` or `~jwst.datamodels.ModelContainer` instance
+        The datamodel or container of datamodels to check
+        
+    cal_step : str
+        The attribute in meta.cal_step to check
+    
+    Returns
+    -------
+    status : str
+        The status of the step in meta.cal_step, typically 'COMPLETE' or 'SKIPPED'
+
+    Notes
+    -----
+    In principle, a step could set the COMPLETE status for only some subset 
+    of models, so checking the zeroth model instance may not always be correct.
+    However, this is not currently done in the pipeline. This function should be
+    updated to accommodate that use-case as needed.
+    """
+    if isinstance(datamodel, Sequence):
+        return getattr(datamodel[0].meta.cal_step, cal_step, NOT_SET)
+    else:
+        return getattr(datamodel.meta.cal_step, cal_step, NOT_SET)
diff --git a/jwst/tweakreg/tweakreg_step.py b/jwst/tweakreg/tweakreg_step.py
index d928cb166..f0d7d6860 100644
--- a/jwst/tweakreg/tweakreg_step.py
+++ b/jwst/tweakreg/tweakreg_step.py
@@ -16,6 +16,7 @@ from tweakwcs.correctors import JWSTWCSCorrector
 from tweakwcs.matchutils import XYXYMatch
 
 from jwst.datamodels import ModelContainer
+from jwst.stpipe import record_step_status
 
 # LOCAL
 from ..stpipe import Step
@@ -204,10 +205,8 @@ class TweakRegStep(Step):
                 self.log.warning("At least two exposures are required for image "
                                  "alignment.")
                 self.log.warning("Nothing to do. Skipping 'TweakRegStep'...")
-                self.skip = True
-                for model in images:
-                    model.meta.cal_step.tweakreg = "SKIPPED"
-                return input
+                record_step_status(images, "tweakreg", success=False)
+                return images
 
         # === start processing images ===
 
@@ -311,10 +310,8 @@ class TweakRegStep(Step):
                     self.log.warning("At least two exposures are required for "
                                      "image alignment.")
                     self.log.warning("Nothing to do. Skipping 'TweakRegStep'...")
-                    for model in images:
-                        model.meta.cal_step.tweakreg = "SKIPPED"
+                    record_step_status(images, "tweakreg", success=False)
                     if not align_to_abs_refcat:
-                        self.skip = True
                         return images
                     local_align_failed = True
                 else:
@@ -329,9 +326,7 @@ class TweakRegStep(Step):
                                    "matched to a single reference source. Try to "
                                    "adjust 'tolerance' and/or 'separation' parameters.")
                     self.log.warning("Skipping 'TweakRegStep'...")
-                    self.skip = True
-                    for model in images:
-                        model.meta.cal_step.tweakreg = "SKIPPED"
+                    record_step_status(images, "tweakreg", success=False)
                     return images
                 else:
                     raise e
@@ -341,9 +336,7 @@ class TweakRegStep(Step):
                     self.log.warning("Skipping relative alignment (stage 1)...")
                 else:
                     self.log.warning("Skipping 'TweakRegStep'...")
-                    self.skip = True
-                    for model in images:
-                        model.meta.cal_step.tweakreg = "SKIPPED"
+                    record_step_status(images, "tweakreg", success=False)
                     return images
 
         if align_to_abs_refcat:
@@ -461,9 +454,7 @@ class TweakRegStep(Step):
                         )
                         if local_align_failed or n_groups == 1:
                             self.log.warning("Nothing to do. Skipping 'TweakRegStep'...")
-                            for model in images:
-                                model.meta.cal_step.tweakreg = "SKIPPED"
-                            self.skip = True
+                            record_step_status(images, "tweakreg", success=False)
                             return images
                     else:
                         raise e
@@ -482,9 +473,7 @@ class TweakRegStep(Step):
                         )
                         if local_align_failed or n_groups == 1:
                             self.log.warning("Skipping 'TweakRegStep'...")
-                            self.skip = True
-                            for model in images:
-                                model.meta.cal_step.tweakreg = "SKIPPED"
+                            record_step_status(images, "tweakreg", success=False)
                             return images
                     else:
                         raise e
diff --git a/pyproject.toml b/pyproject.toml
index 11dcffc07..aa865e519 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -34,7 +34,7 @@ dependencies = [
     "scikit-image>=0.19",
     "scipy>=1.9.3",
     "spherical-geometry>=1.2.22",
-    "stcal>=1.7.1,<1.8.0",
+    "stcal>=1.7.3,<1.8.0",
     "stdatamodels>=2.0.0,<2.1.0",
     "stpipe>=0.6.0,<0.7.0",
     "stsci.image>=2.3.5",


Test patch:
diff --git a/jwst/master_background/tests/test_nirspec_corrections.py b/jwst/master_background/tests/test_nirspec_corrections.py
index 5eaa41fcc..9be59fd1e 100644
--- a/jwst/master_background/tests/test_nirspec_corrections.py
+++ b/jwst/master_background/tests/test_nirspec_corrections.py
@@ -59,7 +59,7 @@ def test_fs_correction():
 
 @pytest.mark.parametrize('name,status',
                          [('BKG101', True), ('bkg101', True),
-                          ('background_101', False), ('101', False),
+                          ('background_101', True), ('101', False),
                           (None, False)])
 def test_is_background(name, status):
     """Test check for background slit."""
diff --git a/jwst/outlier_detection/tests/test_outlier_detection.py b/jwst/outlier_detection/tests/test_outlier_detection.py
index 40982cd4a..af139b5e3 100644
--- a/jwst/outlier_detection/tests/test_outlier_detection.py
+++ b/jwst/outlier_detection/tests/test_outlier_detection.py
@@ -346,18 +346,10 @@ def test_outlier_step_weak_cr_tso(exptype, tsovisit):
     im = we_many_sci(
         numsci=numsci, background=bkg, sigma=sig, signal=signal, exptype=exptype, tsovisit=tsovisit
     )
-    cube_data = np.array([i.data for i in im])
-    cube_err = np.array([i.err for i in im])
-    cube_dq = np.array([i.dq for i in im])
-    cube_var_noise = np.array([i.var_rnoise for i in im])
-    cube = datamodels.CubeModel(data=cube_data, err=cube_err, dq=cube_dq, var_noise=cube_var_noise)
-
-    # update metadata of cube to match the first image
-    cube.meta = im[0].meta
 
     # Drop a weak CR on the science array
     cr_timestep = 5
-    cube.data[cr_timestep, 12, 12] = bkg + sig * 10
+    im[cr_timestep].data[12, 12] = bkg + sig * 10
 
     # make time variability that has larger total amplitude than
     # the CR signal but deviations frame-by-frame are smaller
@@ -366,10 +358,20 @@ def test_outlier_step_weak_cr_tso(exptype, tsovisit):
         model.data[7, 7] += real_time_variability[i]
         model.err[7, 7] = np.sqrt(sig ** 2 + model.data[7, 7])
 
+    cube_data = np.array([i.data for i in im])
+    cube_err = np.array([i.err for i in im])
+    cube_dq = np.array([i.dq for i in im])
+    cube_var_noise = np.array([i.var_rnoise for i in im])
+    cube = datamodels.CubeModel(data=cube_data, err=cube_err, dq=cube_dq, var_noise=cube_var_noise)
+
+    # update metadata of cube to match the first image
+    cube.meta = im[0].meta
+
     result = OutlierDetectionStep.call(cube, rolling_window_width=rolling_window_width)
 
     # Make sure nothing changed in SCI array
-    np.testing.assert_allclose(cube.data, result.data)
+    for i, model in enumerate(im):
+        np.testing.assert_allclose(model.data, result.data[i])
 
     # Verify source is not flagged
     assert np.all(result.dq[:, 7, 7] == datamodels.dqflags.pixel["GOOD"])
diff --git a/jwst/stpipe/tests/test_utilities.py b/jwst/stpipe/tests/test_utilities.py
index 0f3508718..0a1d543d4 100644
--- a/jwst/stpipe/tests/test_utilities.py
+++ b/jwst/stpipe/tests/test_utilities.py
@@ -1,9 +1,11 @@
 """Test utility funcs"""
 from stpipe.utilities import resolve_step_class_alias
+from jwst.stpipe import record_step_status, query_step_status
 
-from jwst.stpipe.utilities import all_steps
+from jwst.stpipe.utilities import all_steps, NOT_SET
 import jwst.pipeline
 import jwst.step
+from jwst import datamodels as dm
 
 
 # All steps available to users should be represented in one
@@ -25,3 +27,30 @@ def test_resolve_step_class_alias():
         if pipeline_class.class_alias is not None:
             assert resolve_step_class_alias(pipeline_class.class_alias) == full_class_name
         assert resolve_step_class_alias(full_class_name) == full_class_name
+
+
+def test_record_query_step_status():
+
+    # single model case
+    model = dm.MultiSpecModel()
+    record_step_status(model, 'test_step', success=True)
+    assert query_step_status(model, 'test_step') == 'COMPLETE'
+
+    # modelcontainer case where all skipped
+    model2 = dm.ModelContainer()
+    model2.append(dm.MultiSpecModel())
+    model2.append(dm.MultiSpecModel())
+    record_step_status(model2, 'test_step', success=False)
+    assert model2[0].meta.cal_step._instance['test_step'] == 'SKIPPED'
+    assert model2[1].meta.cal_step._instance['test_step'] == 'SKIPPED'
+    assert query_step_status(model2, 'test_step') == 'SKIPPED'
+
+    # modelcontainer case with at least one complete
+    # currently the zeroth is checked, so this should return SKIPPED
+    model2.append(dm.MultiSpecModel())
+    model2[2].meta.cal_step._instance['test_step'] = 'COMPLETE'
+    assert query_step_status(model2, 'test_step') == 'SKIPPED'
+
+    # test query not set
+    model3 = dm.MultiSpecModel()
+    assert query_step_status(model3, 'test_step') == NOT_SET