Repository: spacetelescope/jwst

Issue:
Spec2Pipeline.master_background step causing unneeded precache fetching
_Issue [JP-1712](https://jira.stsci.edu/browse/JP-1712) was created on JIRA by [Jonathan Eisenhamer](https://jira.stsci.edu/secure/ViewProfile.jspa?name=eisenham):_

Using the given data, an attempt to retrieve the skipped-step `pathloss` reference file is made. If `master_background` is skipped, the pipeline runs as expected.

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/spacetelescope/jwst/pull/5363?src=pr&el=h1) Report
> Merging [#5363](https://codecov.io/gh/spacetelescope/jwst/pull/5363?src=pr&el=desc) into [master](https://codecov.io/gh/spacetelescope/jwst/commit/fd71cad42826d6971159f1e227662e4156ddb225?el=desc) will **increase** coverage by `0.10%`.
> The diff coverage is `25.00%`.

[![Impacted file tree graph](https://codecov.io/gh/spacetelescope/jwst/pull/5363/graphs/tree.svg?width=650&height=150&src=pr&token=Utf5Zs9g7z)](https://codecov.io/gh/spacetelescope/jwst/pull/5363?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5363      +/-   ##
==========================================
+ Coverage   52.78%   52.88%   +0.10%     
==========================================
  Files         409      409              
  Lines       37030    37098      +68     
  Branches     5746     5760      +14     
==========================================
+ Hits        19545    19618      +73     
+ Misses      16242    16231      -11     
- Partials     1243     1249       +6     
```

| Flag | Coverage Δ | |
|---|---|---|
| #unit | `52.88% <25.00%> (+0.10%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/spacetelescope/jwst/pull/5363?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [jwst/ramp\_fitting/ramp\_fit\_step.py](https://codecov.io/gh/spacetelescope/jwst/pull/5363/diff?src=pr&el=tree#diff-andzdC9yYW1wX2ZpdHRpbmcvcmFtcF9maXRfc3RlcC5weQ==) | `30.95% <0.00%> (ø)` | |
| [jwst/stpipe/pipeline.py](https://codecov.io/gh/spacetelescope/jwst/pull/5363/diff?src=pr&el=tree#diff-andzdC9zdHBpcGUvcGlwZWxpbmUucHk=) | `82.14% <ø> (+15.00%)` | :arrow_up: |
| [jwst/flatfield/flat\_field.py](https://codecov.io/gh/spacetelescope/jwst/pull/5363/diff?src=pr&el=tree#diff-andzdC9mbGF0ZmllbGQvZmxhdF9maWVsZC5weQ==) | `19.78% <50.00%> (ø)` | |
| [jwst/datamodels/util.py](https://codecov.io/gh/spacetelescope/jwst/pull/5363/diff?src=pr&el=tree#diff-andzdC9kYXRhbW9kZWxzL3V0aWwucHk=) | `80.27% <0.00%> (-1.29%)` | :arrow_down: |
| [jwst/resample/resample.py](https://codecov.io/gh/spacetelescope/jwst/pull/5363/diff?src=pr&el=tree#diff-andzdC9yZXNhbXBsZS9yZXNhbXBsZS5weQ==) | `78.49% <0.00%> (-1.06%)` | :arrow_down: |
| [jwst/pipeline/calwebb\_image3.py](https://codecov.io/gh/spacetelescope/jwst/pull/5363/diff?src=pr&el=tree#diff-andzdC9waXBlbGluZS9jYWx3ZWJiX2ltYWdlMy5weQ==) | `35.00% <0.00%> (-0.90%)` | :arrow_down: |
| [jwst/resample/resample\_step.py](https://codecov.io/gh/spacetelescope/jwst/pull/5363/diff?src=pr&el=tree#diff-andzdC9yZXNhbXBsZS9yZXNhbXBsZV9zdGVwLnB5) | `51.51% <0.00%> (-0.53%)` | :arrow_down: |
| [jwst/stpipe/step.py](https://codecov.io/gh/spacetelescope/jwst/pull/5363/diff?src=pr&el=tree#diff-andzdC9zdHBpcGUvc3RlcC5weQ==) | `80.15% <0.00%> (+0.19%)` | :arrow_up: |
| [jwst/outlier\_detection/outlier\_detection\_step.py](https://codecov.io/gh/spacetelescope/jwst/pull/5363/diff?src=pr&el=tree#diff-andzdC9vdXRsaWVyX2RldGVjdGlvbi9vdXRsaWVyX2RldGVjdGlvbl9zdGVwLnB5) | `40.33% <0.00%> (+0.50%)` | :arrow_up: |
| [jwst/lib/basic\_utils.py](https://codecov.io/gh/spacetelescope/jwst/pull/5363/diff?src=pr&el=tree#diff-andzdC9saWIvYmFzaWNfdXRpbHMucHk=) | `76.47% <0.00%> (+9.80%)` | :arrow_up: |
| ... and [1 more](https://codecov.io/gh/spacetelescope/jwst/pull/5363/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/spacetelescope/jwst/pull/5363?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/spacetelescope/jwst/pull/5363?src=pr&el=footer). Last update [877e72b...9cdf16e](https://codecov.io/gh/spacetelescope/jwst/pull/5363?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index 0584b80df..4b8b1e1a3 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -105,6 +105,11 @@ scripts
 
 - Add pointing analysis commands v1_calculate and pointing_summary. [#5311]
 
+stpipe
+------
+
+- Do not attempt prefetch on pipelines that are set to not allow prefetch. [#5363]
+
 ramp_fitting
 ------------
 
diff --git a/jwst/stpipe/pipeline.py b/jwst/stpipe/pipeline.py
index 5d29bbcf4..a754ac142 100644
--- a/jwst/stpipe/pipeline.py
+++ b/jwst/stpipe/pipeline.py
@@ -85,9 +85,12 @@ class Pipeline(Step):
 
     @property
     def _unskipped_steps(self):
-        """Return a list of the unskipped Step objects launched by `self`."""
+        """Return a list of the unskipped Step objects launched by `self`.
+
+        Steps are also excluded if `Step.prefetch_references` is False.
+        """
         return [getattr(self, name) for name in self.step_defs
-                if not getattr(self, name).skip]
+                if (not getattr(self, name).skip and getattr(self, name).prefetch_references)]
 
     def get_ref_override(self, reference_file_type):
         """Return any override for `reference_file_type` for any of the steps in


Test patch:
diff --git a/jwst/stpipe/tests/steps/__init__.py b/jwst/stpipe/tests/steps/__init__.py
index 7e5955254..865c87de8 100644
--- a/jwst/stpipe/tests/steps/__init__.py
+++ b/jwst/stpipe/tests/steps/__init__.py
@@ -5,6 +5,28 @@ from jwst.datamodels import (
     ModelContainer,
 )
 
+class StepWithReference(Step):
+    """Step that refers to a reference file"""
+
+    reference_file_types = ['flat']
+
+    def process(self, data):
+        return data
+
+
+class PipeWithReference(Pipeline):
+    """Pipeline calling step with reference"""
+
+    spec = """
+    """
+
+    step_defs = {'step_with_reference': StepWithReference}
+
+    def process(self, data):
+        result = self.step_with_reference(data)
+
+        return result
+
 
 class AnotherDummyStep(Step):
     """
diff --git a/jwst/stpipe/tests/test_pipeline.py b/jwst/stpipe/tests/test_pipeline.py
index 9e34c0ed3..759a009ad 100644
--- a/jwst/stpipe/tests/test_pipeline.py
+++ b/jwst/stpipe/tests/test_pipeline.py
@@ -5,9 +5,10 @@ import numpy as np
 from numpy.testing import assert_allclose
 import pytest
 
-from jwst.stpipe import Step, Pipeline, LinearPipeline
 from jwst import datamodels
-# TODO: Test system call steps
+from jwst.stpipe import Step, Pipeline, LinearPipeline, crds_client
+
+from .steps import PipeWithReference, StepWithReference
 
 
 def library_function():
@@ -136,6 +137,46 @@ class MyLinearPipeline(LinearPipeline):
         ]
 
 
+def test_prefetch(_jail, monkeypatch):
+    """Test prefetching"""
+
+    # Setup mock to crds to flag if the call was made.
+    class MockGetRef:
+        called = False
+
+        def mock(self, dataset_model, reference_file_types, observatory=None):
+            if 'flat' in reference_file_types:
+                self.called = True
+            result = {
+                reftype: 'N/A'
+                for reftype in reference_file_types
+            }
+            return result
+        __call__ = mock
+    mock_get_ref = MockGetRef()
+    monkeypatch.setattr(crds_client, 'get_multiple_reference_paths', mock_get_ref)
+
+    # Create some data
+    model = datamodels.ImageModel((19, 19))
+    model.meta.instrument.name = "NIRCAM"
+    model.meta.instrument.detector = 'NRCA1'
+    model.meta.instrument.filter = "F070W"
+    model.meta.instrument.pupil = 'CLEAR'
+    model.meta.observation.date = "2019-01-01"
+    model.meta.observation.time = "00:00:00"
+
+    # Run the pipeline with prefetch set.
+    StepWithReference.prefetch_references = True
+    PipeWithReference.call(model)
+    assert mock_get_ref.called
+
+    # Now run with prefetch unset.
+    mock_get_ref.called = False
+    StepWithReference.prefetch_references = False
+    PipeWithReference.call(model)
+    assert not mock_get_ref.called
+
+
 def test_partial_pipeline(_jail):
     pipe = MyLinearPipeline()