Repository: spacetelescope/romancal
Pull request #1664: add regtest resource tracking
Author: login: braingram
Labels: name: Dark Current
name: Photom
name: ramp_fitting
name: Saturation
name: linearity
name: pipeline
name: dq_init
name: regression_testing
name: Wide Field Instrument (WFI)
Created: 2025-03-24T19:27:41Z | Merged: 2025-03-26T14:15:43Z

This PR adds test fixtures to make it easier for regtests to log peak memory usage and runtime (specifically of fixtures) to allow using regtests runs as pseudo benchmarks. The general API is:
```python
@pytest.fixture
def my_long_fixture(resource_tracker):
    with resource_tracker.track():
        # something that takes memory and time
        pass

def test_log_tracked_resources(log_tracked_resources, my_long_fixture):
    log_tracked_resources()
```
or for a test where the long-running or resource-intensive operation is in the test (not a fixture):
```python
def test_long_step(resource_tracker, request):
    with resource_tracker.track(log=request):
        # something that takes memory and time
        pass
```

At the moment the `resource_tracker` fixture logs peak memory usage and runtime and reports these values in the `report*.xml` generated during regtests run (which is uploaded to github and artifactory).

Regtests: https://github.com/spacetelescope/RegressionTests/actions/runs/14042952366
all pass and generate a report:
https://github.com/spacetelescope/RegressionTests/actions/runs/14042952366/artifacts/2818137935
that contains the recorded resource usage:
```xml
<testcase classname="romancal.regtest.test_mos_skycell_pipeline" name="test_log_tracked_resources" time="500.187"><properties><property name="tracked-peakmem" value="4632279697" /><property name="tracked-time" value="479.6778707471" /></properties>
```

With these included in the results xml combined with the recorded commit hash and dependencies in the snapshot yaml uploaded to artifactory for each regtest run we hopefully have enough information compare performance as changes are made (both from dependencies and from PRs). We would likely benefit from expanding some tooling (like [pytestresultsdiff](https://github.com/zacharyburnett/pytestresultsdiff)) around this to say A-B compare the current PR run with the most recent nightly run.

<!-- if you can't perform these due to permissions, please ask a maintainer to do them -->
## Tasks
- [ ] **request a review from someone specific**, to avoid making the maintainers review every PR
- [ ] add a build milestone, i.e. `24Q4_B15` (use the [latest build](https://github.com/spacetelescope/romancal/milestones) if not sure)
- [ ] Does this PR change user-facing code / API? (if not, label with `no-changelog-entry-needed`)
  - [ ] write news fragment(s) in `changes/`: `echo "changed something" > changes/<PR#>.<changetype>.rst` (see below for change types)
  - [ ] update or add relevant tests
  - [ ] update relevant docstrings and / or `docs/` page
  - [ ] [start a regression test](https://github.com/spacetelescope/RegressionTests/actions/workflows/romancal.yml) and include a link to the running job ([click here for instructions](https://github.com/spacetelescope/RegressionTests/blob/main/docs/running_regression_tests.md))
    - [ ] Do truth files need to be updated ("okified")?
      - [ ] **after the reviewer has approved these changes**, run `okify_regtests` to update the truth files
- [ ] if a JIRA ticket exists, [make sure it is resolved properly](https://github.com/spacetelescope/romancal/wiki/How-to-resolve-JIRA-issues)

<details><summary>news fragment change types...</summary>

  - ``changes/<PR#>.general.rst``: infrastructure or miscellaneous change
  - ``changes/<PR#>.docs.rst``
  - ``changes/<PR#>.stpipe.rst``
  - ``changes/<PR#>.associations.rst``
  - ``changes/<PR#>.scripts.rst``
  - ``changes/<PR#>.mosaic_pipeline.rst``
  - ``changes/<PR#>.patch_match.rst``

  ## steps
  - ``changes/<PR#>.dq_init.rst``
  - ``changes/<PR#>.saturation.rst``
  - ``changes/<PR#>.refpix.rst``
  - ``changes/<PR#>.linearity.rst``
  - ``changes/<PR#>.dark_current.rst``
  - ``changes/<PR#>.jump_detection.rst``
  - ``changes/<PR#>.ramp_fitting.rst``
  - ``changes/<PR#>.assign_wcs.rst``
  - ``changes/<PR#>.flatfield.rst``
  - ``changes/<PR#>.photom.rst``
  - ``changes/<PR#>.flux.rst``
  - ``changes/<PR#>.source_detection.rst``
  - ``changes/<PR#>.tweakreg.rst``
  - ``changes/<PR#>.skymatch.rst``
  - ``changes/<PR#>.outlier_detection.rst``
  - ``changes/<PR#>.resample.rst``
  - ``changes/<PR#>.source_catalog.rst``
</details>



Diff:
diff --git a/changes/1664.general.rst b/changes/1664.general.rst
new file mode 100644
index 000000000..7f0835a9c
--- /dev/null
+++ b/changes/1664.general.rst
@@ -0,0 +1 @@
+Add resource tracking fixtures for regression tests.
diff --git a/romancal/regtest/conftest.py b/romancal/regtest/conftest.py
index bfa12bc26..eb2735009 100644
--- a/romancal/regtest/conftest.py
+++ b/romancal/regtest/conftest.py
@@ -10,6 +10,7 @@
 from numpy.testing import assert_allclose, assert_equal
 
 from romancal.regtest.regtestdata import RegtestData
+from romancal.regtest.resource_tracker import ResourceTracker
 
 TODAYS_DATE = datetime.now().strftime("%Y-%m-%d")
 
@@ -233,6 +234,52 @@ def ignore_asdf_paths():
     return {"ignore": ignore_attr}
 
 
+@pytest.fixture(scope="module")
+def resource_tracker():
+    """Fixture to return the current module-scoped ResourceTracker.
+
+    Use by calling ``track`` to generate a context in which resource
+    usage will be tracked.
+
+    >>>
+    >> with resource_tracker.track():
+    >>     # do stuff
+
+    For resources used during tests providing a function-scoped
+    request fixture result as the log argument will also log the
+    used resources to the junit results.xml.
+
+    >>>
+    >> def test_something(resource_tracker, request):
+    >>     with resource_tracker.track(log=request):
+    >>         # do stuff
+
+    For resources used during fixtures the tracked resources
+    can be logged in a separate test using ``log_tracked_resources``.
+    """
+    return ResourceTracker()
+
+
+@pytest.fixture()
+def log_tracked_resources(resource_tracker, request):
+    """Fixture to log resources tracked by ``resource_tracker``.
+
+    >>>
+    >> @pytest.fixture
+    >> def my_fixture(resource_tracker):
+    >>     with resource_tracker.track():
+    >>         # do stuff
+    >>
+    >> def test_write_log(log_tracked_resources, my_fixture):
+    >>     log_tracked_resources()
+    """
+
+    def callback():
+        resource_tracker.log(request)
+
+    yield callback
+
+
 @pytest.fixture
 def diff_astropy_tables():
     """Compare astropy tables with tolerances for float columns."""
diff --git a/romancal/regtest/resource_tracker.py b/romancal/regtest/resource_tracker.py
new file mode 100644
index 000000000..da4955058
--- /dev/null
+++ b/romancal/regtest/resource_tracker.py
@@ -0,0 +1,110 @@
+"""Resource tracking regtest utilities
+
+Can be used within module-scoped fixtures (often used to
+run Steps or Pipelines) or within tests.
+
+For uses where the resource usage occurs within a test:
+
+>>>
+>> def test_long_step(resource_tracker, request):
+>>     with resource_tracker.track(log=request):
+>>         # something that takes memory and time
+>>         pass
+
+For a module-scoped fixture the resource tracking can
+be performed in the fixture but the logging/reporting of
+the resource usage must occur during a test:
+
+>>>
+>> @pytest.fixture(scope="module")
+>> def resource_tracker():
+>>     return ResourceTracker()
+>>
+>> @pytest.fixture()
+>> def log_tracked_resources(resource_tracker, request):
+>>     def callback():
+>>         resource_tracker.log(request)
+>>
+>>     yield callback
+>>
+>> @pytest.fixture
+>> def my_long_fixture(resource_tracker):
+>>     with resource_tracker.track():
+>>         # something that takes memory and time
+>>         pass
+>>
+>> def test_log_tracked_resources(log_tracked_resources, my_long_fixture):
+>>     log_tracked_resources()
+
+Use of the module-scoped fixture has fixture-reuse
+considerations similar to the ``rtdata_module`` fixture. Having
+more than one module scoped fixture that uses ``resource_tracker``
+per module is discouraged (as both will use the same ``ResourceTracker``
+instance). Parameterization of a fixture using ``resource_tracker``
+is supported (same as ``rtdata_module``).
+"""
+
+import time
+import tracemalloc
+from contextlib import ExitStack, contextmanager
+
+
+class TrackRuntime:
+    """Runtime tracker context."""
+
+    def __enter__(self):
+        self._t0 = time.monotonic()
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.value = time.monotonic() - self._t0
+
+    def log(self):
+        return ("tracked-time", self.value)
+
+
+class TrackPeakMemory:
+    """Peak memory tracker context."""
+
+    def __enter__(self):
+        tracemalloc.start()
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        _, self.value = tracemalloc.get_traced_memory()
+        tracemalloc.stop()
+
+    def log(self):
+        return ("tracked-peakmem", self.value)
+
+
+class ResourceTracker:
+    """Track resources used during track context."""
+
+    def __init__(self):
+        self._trackers = [TrackPeakMemory(), TrackRuntime()]
+
+    def log(self, request):
+        """Log tracked resource usage to the pytest request user properties.
+
+        Parameters
+        ----------
+        request : pytest.FixtureRequest
+            Must be a function-scoped pytest request fixture result.
+        """
+        request.node.user_properties.extend(t.log() for t in self._trackers)
+
+    @contextmanager
+    def track(self, log=None):
+        """Context during which resources are tracked.
+
+        Parameters
+        ----------
+        log : pytest.FixtureRequest, optional
+            If provided, log the usage to the provided request fixture result.
+        """
+        try:
+            with ExitStack() as stack:
+                [stack.enter_context(t) for t in self._trackers]
+                yield self
+        finally:
+            if log:
+                self.log(log)
diff --git a/romancal/regtest/test_catalog.py b/romancal/regtest/test_catalog.py
index 229b811a2..8b22403eb 100644
--- a/romancal/regtest/test_catalog.py
+++ b/romancal/regtest/test_catalog.py
@@ -19,7 +19,7 @@
     ],
     ids=["L3", "L2", "L3skycell"],
 )
-def run_source_catalog(rtdata_module, request):
+def run_source_catalog(rtdata_module, request, resource_tracker):
     rtdata = rtdata_module
 
     inputfn = request.param
@@ -35,7 +35,8 @@ def run_source_catalog(rtdata_module, request):
         "romancal.step.SourceCatalogStep",
         rtdata.input,
     ]
-    RomanStep.from_cmdline(args)
+    with resource_tracker.track():
+        RomanStep.from_cmdline(args)
     return rtdata_module
 
 
@@ -68,6 +69,10 @@ def test_has_field(fields, field):
     assert field in fields
 
 
+def test_log_tracked_resources(log_tracked_resources, run_source_catalog):
+    log_tracked_resources()
+
+
 def test_forced_catalog(rtdata_module):
     rtdata = rtdata_module
     input_deep_segm = "r0099101001001001001_r274dp63x31y81_prompt_F158_segm.asdf"
diff --git a/romancal/regtest/test_dark_current.py b/romancal/regtest/test_dark_current.py
index 2e44c44f0..4511d719e 100644
--- a/romancal/regtest/test_dark_current.py
+++ b/romancal/regtest/test_dark_current.py
@@ -8,7 +8,9 @@
 
 
 @pytest.mark.bigdata
-def test_dark_current_subtraction_step(rtdata, ignore_asdf_paths):
+def test_dark_current_subtraction_step(
+    rtdata, ignore_asdf_paths, resource_tracker, request
+):
     """Function to run and compare Dark Current subtraction files. Note: This
     should include tests for overrides etc."""
 
@@ -17,7 +19,8 @@ def test_dark_current_subtraction_step(rtdata, ignore_asdf_paths):
     rtdata.input = input_datafile
 
     args = ["romancal.step.DarkCurrentStep", rtdata.input]
-    RomanStep.from_cmdline(args)
+    with resource_tracker.track(log=request):
+        RomanStep.from_cmdline(args)
     output = "r0000101001001001001_0001_wfi01_darkcurrent.asdf"
     rtdata.output = output
     rtdata.get_truth(f"truth/WFI/image/{output}")
@@ -26,7 +29,9 @@ def test_dark_current_subtraction_step(rtdata, ignore_asdf_paths):
 
 
 @pytest.mark.bigdata
-def test_dark_current_outfile_step(rtdata, ignore_asdf_paths):
+def test_dark_current_outfile_step(
+    rtdata, ignore_asdf_paths, resource_tracker, request
+):
     """Function to run and compare Dark Current subtraction files. Here the
     test is for renaming the output file."""
     input_datafile = "r0000101001001001001_0001_wfi01_linearity.asdf"
@@ -38,7 +43,8 @@ def test_dark_current_outfile_step(rtdata, ignore_asdf_paths):
         rtdata.input,
         "--output_file=Test_darkcurrent",
     ]
-    RomanStep.from_cmdline(args)
+    with resource_tracker.track(log=request):
+        RomanStep.from_cmdline(args)
     output = "Test_darkcurrent.asdf"
     rtdata.output = output
     rtdata.get_truth(f"truth/WFI/image/{output}")
@@ -47,7 +53,9 @@ def test_dark_current_outfile_step(rtdata, ignore_asdf_paths):
 
 
 @pytest.mark.bigdata
-def test_dark_current_outfile_suffix(rtdata, ignore_asdf_paths):
+def test_dark_current_outfile_suffix(
+    rtdata, ignore_asdf_paths, resource_tracker, request
+):
     """Function to run and compare Dark Current subtraction files. Here the
     test is for renaming the output file."""
     input_datafile = "r0000101001001001001_0001_wfi01_linearity.asdf"
@@ -60,7 +68,8 @@ def test_dark_current_outfile_suffix(rtdata, ignore_asdf_paths):
         "--output_file=Test_dark",
         '--suffix="suffix_test"',
     ]
-    RomanStep.from_cmdline(args)
+    with resource_tracker.track(log=request):
+        RomanStep.from_cmdline(args)
     output = "Test_darkcurrent.asdf"
     rtdata.output = output
     rtdata.get_truth(f"truth/WFI/image/{output}")
@@ -69,7 +78,7 @@ def test_dark_current_outfile_suffix(rtdata, ignore_asdf_paths):
 
 
 @pytest.mark.bigdata
-def test_dark_current_output(rtdata, ignore_asdf_paths):
+def test_dark_current_output(rtdata, ignore_asdf_paths, resource_tracker, request):
     """Function to run and compare Dark Current subtraction files. Here the
     test for overriding the CRDS dark reference file."""
 
@@ -83,7 +92,8 @@ def test_dark_current_output(rtdata, ignore_asdf_paths):
         rtdata.input,
         f"--dark_output={dark_output_name}",
     ]
-    RomanStep.from_cmdline(args)
+    with resource_tracker.track(log=request):
+        RomanStep.from_cmdline(args)
     output = "r0000101001001001001_0001_wfi01_darkcurrent.asdf"
     rtdata.output = output
     rtdata.get_truth(f"truth/WFI/image/{output}")
diff --git a/romancal/regtest/test_linearity.py b/romancal/regtest/test_linearity.py
index 6dc459af4..47355e175 100644
--- a/romancal/regtest/test_linearity.py
+++ b/romancal/regtest/test_linearity.py
@@ -8,14 +8,15 @@
 
 
 @pytest.mark.bigdata
-def test_linearity_step(rtdata, ignore_asdf_paths):
+def test_linearity_step(rtdata, ignore_asdf_paths, resource_tracker, request):
     """Function to run and compare linearity correction files."""
     input_file = "r0000101001001001001_0001_wfi01_refpix.asdf"
     rtdata.get_data(f"WFI/image/{input_file}")
     rtdata.input = input_file
 
     args = ["romancal.step.LinearityStep", rtdata.input]
-    RomanStep.from_cmdline(args)
+    with resource_tracker.track(log=request):
+        RomanStep.from_cmdline(args)
     output = "r0000101001001001001_0001_wfi01_linearity.asdf"
     rtdata.output = output
     rtdata.get_truth(f"truth/WFI/image/{output}")
@@ -24,7 +25,7 @@ def test_linearity_step(rtdata, ignore_asdf_paths):
 
 
 @pytest.mark.bigdata
-def test_linearity_outfile_step(rtdata, ignore_asdf_paths):
+def test_linearity_outfile_step(rtdata, ignore_asdf_paths, resource_tracker, request):
     """Function to run and linearity correction files. Here the
     test is for renaming the output file."""
     input_file = "r0000101001001001001_0001_wfi01_refpix.asdf"
@@ -32,7 +33,8 @@ def test_linearity_outfile_step(rtdata, ignore_asdf_paths):
     rtdata.input = input_file
 
     args = ["romancal.step.LinearityStep", rtdata.input, "--output_file=Test_linearity"]
-    RomanStep.from_cmdline(args)
+    with resource_tracker.track(log=request):
+        RomanStep.from_cmdline(args)
     output = "Test_linearity.asdf"
     rtdata.output = output
     rtdata.get_truth(f"truth/WFI/image/{output}")
diff --git a/romancal/regtest/test_mos_pipeline.py b/romancal/regtest/test_mos_pipeline.py
index 06445d1d0..bd948a369 100644
--- a/romancal/regtest/test_mos_pipeline.py
+++ b/romancal/regtest/test_mos_pipeline.py
@@ -16,7 +16,7 @@
 
 
 @pytest.fixture(scope="module")
-def run_mos(rtdata_module):
+def run_mos(rtdata_module, resource_tracker):
     rtdata = rtdata_module
 
     rtdata.get_asn("WFI/image/L3_regtest_asn.json")
@@ -29,7 +29,8 @@ def run_mos(rtdata_module):
         "roman_mos",
         rtdata.input,
     ]
-    MosaicPipeline.from_cmdline(args)
+    with resource_tracker.track():
+        MosaicPipeline.from_cmdline(args)
 
     rtdata.get_truth(f"truth/WFI/image/{output}")
     return rtdata
@@ -67,6 +68,10 @@ def preview_filename(output_filename):
     return preview_filename
 
 
+def test_log_tracked_resources(log_tracked_resources, run_mos):
+    log_tracked_resources()
+
+
 def test_output_matches_truth(output_filename, truth_filename, ignore_asdf_paths):
     # DMS356
     diff = compare_asdf(output_filename, truth_filename, **ignore_asdf_paths)
diff --git a/romancal/regtest/test_mos_skycell_pipeline.py b/romancal/regtest/test_mos_skycell_pipeline.py
index 016708ee2..9ac9db795 100644
--- a/romancal/regtest/test_mos_skycell_pipeline.py
+++ b/romancal/regtest/test_mos_skycell_pipeline.py
@@ -12,7 +12,7 @@
 
 
 @pytest.fixture(scope="module")
-def run_mos(rtdata_module):
+def run_mos(rtdata_module, resource_tracker):
     rtdata = rtdata_module
 
     # Test Pipeline
@@ -23,7 +23,8 @@ def run_mos(rtdata_module):
         "roman_mos",
         rtdata.input,
     ]
-    MosaicPipeline.from_cmdline(args)
+    with resource_tracker.track():
+        MosaicPipeline.from_cmdline(args)
     rtdata.get_truth(f"truth/WFI/image/{output}")
     return rtdata
 
@@ -44,6 +45,10 @@ def truth_filename(run_mos):
     return run_mos.truth
 
 
+def test_log_tracked_resources(log_tracked_resources, run_mos):
+    log_tracked_resources()
+
+
 def test_output_matches_truth(output_filename, truth_filename, ignore_asdf_paths):
     diff = compare_asdf(output_filename, truth_filename, **ignore_asdf_paths)
     assert diff.identical, diff.report()
diff --git a/romancal/regtest/test_multiband_catalog.py b/romancal/regtest/test_multiband_catalog.py
index 03a8ef58f..a0c0f5753 100644
--- a/romancal/regtest/test_multiband_catalog.py
+++ b/romancal/regtest/test_multiband_catalog.py
@@ -29,7 +29,7 @@
 ]
 
 
-def test_multiband_catalog(rtdata_module):
+def test_multiband_catalog(rtdata_module, resource_tracker, request):
     rtdata = rtdata_module
     inputasnfn = "L3_skycell_mbcat_asn.json"
     # note that this input association currently only has a single
@@ -48,7 +48,8 @@ def test_multiband_catalog(rtdata_module):
         "--deblend",
         "True",  # use deblending, DMS 393
     ]
-    RomanStep.from_cmdline(args)
+    with resource_tracker.track(log=request):
+        RomanStep.from_cmdline(args)
     afcat = asdf.open(outputfn)
     for field in fieldlist:
         assert field in afcat["roman"]["source_catalog"].dtype.names
diff --git a/romancal/regtest/test_ramp_fitting.py b/romancal/regtest/test_ramp_fitting.py
index 5e8f78842..76c033d13 100644
--- a/romancal/regtest/test_ramp_fitting.py
+++ b/romancal/regtest/test_ramp_fitting.py
@@ -141,7 +141,7 @@ def cond_science_verification(
     ],
     ids=["image_full", "spec_full", "image_trunc", "spec_trunc"],
 )
-def rampfit_result(request, rtdata_module):
+def rampfit_result(request, rtdata_module, resource_tracker):
     """Run RampFitStep
 
     Parameters
@@ -160,7 +160,8 @@ def rampfit_result(request, rtdata_module):
     rtdata_module.input = input_data
 
     # Execute the step
-    result_model = RampFitStep.call(input_data, save_results=True)
+    with resource_tracker.track():
+        result_model = RampFitStep.call(input_data, save_results=True)
 
     # Setup outputs
     input_data_path = Path(input_data)
@@ -181,6 +182,11 @@ def rampfit_result(request, rtdata_module):
 # #####
 # Tests
 # #####
+@pytest.mark.bigdata
+def test_log_tracked_resources(log_tracked_resources, rampfit_result):
+    log_tracked_resources()
+
+
 @pytest.mark.bigdata
 def test_rampfit_step(rampfit_result, rtdata_module, ignore_asdf_paths):
     """Test rampfit result against various conditions"""
diff --git a/romancal/regtest/test_refpix.py b/romancal/regtest/test_refpix.py
index 02eff9cb9..8a4615a80 100644
--- a/romancal/regtest/test_refpix.py
+++ b/romancal/regtest/test_refpix.py
@@ -8,14 +8,15 @@
 
 
 @pytest.mark.bigdata
-def test_refpix_step(rtdata, ignore_asdf_paths):
+def test_refpix_step(rtdata, ignore_asdf_paths, resource_tracker, request):
     # I have no idea what this is supposed to be
     input_datafile = "r0000101001001001001_0001_wfi01_saturation.asdf"
     rtdata.get_data(f"WFI/image/{input_datafile}")
     rtdata.input = input_datafile
 
     args = ["romancal.step.RefPixStep", rtdata.input]
-    RomanStep.from_cmdline(args)
+    with resource_tracker.track(log=request):
+        RomanStep.from_cmdline(args)
 
     # Again I have no idea here
     output = "r0000101001001001001_0001_wfi01_refpix.asdf"
diff --git a/romancal/regtest/test_resample.py b/romancal/regtest/test_resample.py
index 1b1e5345b..df3ae18a6 100644
--- a/romancal/regtest/test_resample.py
+++ b/romancal/regtest/test_resample.py
@@ -9,7 +9,7 @@
 
 
 @pytest.mark.bigdata
-def test_resample_single_file(rtdata, ignore_asdf_paths):
+def test_resample_single_file(rtdata, ignore_asdf_paths, resource_tracker, request):
     output_data = "mosaic_resamplestep.asdf"
 
     rtdata.get_asn("WFI/image/L3_mosaic_asn.json")
@@ -27,7 +27,9 @@ def test_resample_single_file(rtdata, ignore_asdf_paths):
         "--resample_on_skycell=False",
         f"--output_file='{rtdata.output}'",
     ]
-    RomanStep.from_cmdline(args)
+    with resource_tracker.track(log=request):
+        RomanStep.from_cmdline(args)
+
     resample_out = rdm.open(rtdata.output)
 
     step.log.info(
diff --git a/romancal/regtest/test_resource_tracker.py b/romancal/regtest/test_resource_tracker.py
new file mode 100644
index 000000000..b7eca404a
--- /dev/null
+++ b/romancal/regtest/test_resource_tracker.py
@@ -0,0 +1,56 @@
+import os
+import time
+
+from romancal.regtest.resource_tracker import (
+    ResourceTracker,
+    TrackPeakMemory,
+    TrackRuntime,
+)
+
+
+class FakeNode:
+    def __init__(self):
+        self.user_properties = []
+
+
+class FakeRequest:
+    def __init__(self):
+        self.node = FakeNode()
+
+
+def test_runtime():
+    tracker = TrackRuntime()
+    with tracker:
+        time.sleep(1.0)
+    # a 1 second sleep is sometimes too much to ask
+    # of CI runners. Use a wide margin to make this test
+    # less brittle in those cases.
+    threshold = 0.5 if "CI" in os.environ else 0.1
+    assert abs(tracker.log()[1] - 1.0) < threshold
+
+
+def test_memory():
+    tracker = TrackPeakMemory()
+    N = 1024 * 1024
+    with tracker:
+        b = b"0" * N  # noqa: F841
+    assert abs(tracker.log()[1] - N) / N < 0.01
+
+
+def test_resource_tracker():
+    tracker = ResourceTracker()
+    with tracker.track():
+        pass
+    fake_request = FakeRequest()
+    tracker.log(fake_request)
+    keys = {log[0] for log in fake_request.node.user_properties}
+    assert keys == {"tracked-time", "tracked-peakmem"}
+
+
+def test_log():
+    tracker = ResourceTracker()
+    fake_request = FakeRequest()
+    with tracker.track(log=fake_request):
+        pass
+    keys = {log[0] for log in fake_request.node.user_properties}
+    assert keys == {"tracked-time", "tracked-peakmem"}
diff --git a/romancal/regtest/test_tweakreg.py b/romancal/regtest/test_tweakreg.py
index 07c335e7f..15a5ba5d3 100644
--- a/romancal/regtest/test_tweakreg.py
+++ b/romancal/regtest/test_tweakreg.py
@@ -11,7 +11,7 @@
 
 
 @pytest.mark.bigdata
-def test_tweakreg(rtdata, ignore_asdf_paths, tmp_path):
+def test_tweakreg(rtdata, ignore_asdf_paths, tmp_path, resource_tracker, request):
     # N.B.: uncal file is from simulator
     # ``shifted'' version is created in make_regtestdata.sh; cal file is taken,
     # the wcsinfo is perturbed, and AssignWCS is run to update the WCS with the
@@ -39,7 +39,9 @@ def test_tweakreg(rtdata, ignore_asdf_paths, tmp_path):
         f"--output_file='{rtdata.output}'",
         "--suffix='tweakregstep'",
     ]
-    RomanStep.from_cmdline(args)
+    with resource_tracker.track(log=request):
+        RomanStep.from_cmdline(args)
+
     tweakreg_out = rdm.open(rtdata.output)
 
     step.log.info(
diff --git a/romancal/regtest/test_wfi_dq_init.py b/romancal/regtest/test_wfi_dq_init.py
index 4e779f171..f526bc8d8 100644
--- a/romancal/regtest/test_wfi_dq_init.py
+++ b/romancal/regtest/test_wfi_dq_init.py
@@ -12,7 +12,7 @@
 
 
 @pytest.mark.bigdata
-def test_dq_init_image_step(rtdata, ignore_asdf_paths):
+def test_dq_init_image_step(rtdata, ignore_asdf_paths, resource_tracker, request):
     """DMS25 Test: Testing retrieval of best ref file for image data,
     and creation of a ramp file with CRDS selected mask file applied."""
 
@@ -48,7 +48,9 @@ def test_dq_init_image_step(rtdata, ignore_asdf_paths):
         " The first ERROR is expected, due to extra CRDS parameters"
         " not having been implemented yet."
     )
-    RomanStep.from_cmdline(args)
+    with resource_tracker.track(log=request):
+        RomanStep.from_cmdline(args)
+
     ramp_out = rdm.open(rtdata.output)
     step.log.info(
         "DMS25 MSG: Does ramp data contain pixeldq from mask file? :"
@@ -67,7 +69,7 @@ def test_dq_init_image_step(rtdata, ignore_asdf_paths):
 
 
 @pytest.mark.bigdata
-def test_dq_init_grism_step(rtdata, ignore_asdf_paths):
+def test_dq_init_grism_step(rtdata, ignore_asdf_paths, resource_tracker, request):
     """DMS25 Test: Testing retrieval of best ref file for grism data,
     and creation of a ramp file with CRDS selected mask file applied."""
 
@@ -103,7 +105,9 @@ def test_dq_init_grism_step(rtdata, ignore_asdf_paths):
         "The first ERROR is expected, due to extra CRDS parameters "
         "not having been implemented yet."
     )
-    RomanStep.from_cmdline(args)
+    with resource_tracker.track(log=request):
+        RomanStep.from_cmdline(args)
+
     ramp_out = rdm.open(rtdata.output)
     step.log.info(
         "DMS25 MSG: Does ramp data contain pixeldq from mask file? :"
diff --git a/romancal/regtest/test_wfi_flat_field.py b/romancal/regtest/test_wfi_flat_field.py
index 0884acd46..70763460d 100644
--- a/romancal/regtest/test_wfi_flat_field.py
+++ b/romancal/regtest/test_wfi_flat_field.py
@@ -11,7 +11,7 @@
 
 
 @pytest.mark.bigdata
-def test_flat_field_image_step(rtdata, ignore_asdf_paths):
+def test_flat_field_image_step(rtdata, ignore_asdf_paths, resource_tracker, request):
     """Test for the flat field step using imaging data."""
 
     input_data = "r0000101001001001001_0001_wfi01_assignwcs.asdf"
@@ -28,15 +28,16 @@ def test_flat_field_image_step(rtdata, ignore_asdf_paths):
     output = "r0000101001001001001_0001_wfi01_flat.asdf"
     rtdata.output = output
     args = ["romancal.step.FlatFieldStep", rtdata.input]
-    RomanStep.from_cmdline(args)
+    with resource_tracker.track(log=request):
+        RomanStep.from_cmdline(args)
+
     rtdata.get_truth(f"truth/WFI/image/{output}")
     diff = compare_asdf(rtdata.output, rtdata.truth, **ignore_asdf_paths)
     assert diff.identical, diff.report()
 
 
-# @pytest.mark.xfail(reason="Wrong input for the purpose of this test")
 @pytest.mark.bigdata
-def test_flat_field_grism_step(rtdata, ignore_asdf_paths):
+def test_flat_field_grism_step(rtdata, ignore_asdf_paths, resource_tracker, request):
     """Test for the flat field step using grism data. The reference file for
     the grism and prism data should be None, only testing the grism
     case here."""
@@ -63,7 +64,9 @@ def test_flat_field_grism_step(rtdata, ignore_asdf_paths):
     output = "r0000201001001001001_0001_wfi01_flat.asdf"
     rtdata.output = output
     args = ["romancal.step.FlatFieldStep", rtdata.input]
-    RomanStep.from_cmdline(args)
+    with resource_tracker.track(log=request):
+        RomanStep.from_cmdline(args)
+
     output_model = rdm.open(rtdata.output)
     assert output_model.meta.cal_step.flat_field == "SKIPPED"
     rtdata.get_truth(f"truth/WFI/grism/{output}")
@@ -73,7 +76,9 @@ def test_flat_field_grism_step(rtdata, ignore_asdf_paths):
 
 @pytest.mark.bigdata
 @pytest.mark.soctests
-def test_flat_field_crds_match_image_step(rtdata, ignore_asdf_paths):
+def test_flat_field_crds_match_image_step(
+    rtdata, ignore_asdf_paths, resource_tracker, request
+):
     """DMS79 Test: Testing that different datetimes pull different
     flat files and successfully make level 2 output"""
 
@@ -113,7 +118,9 @@ def test_flat_field_crds_match_image_step(rtdata, ignore_asdf_paths):
         "expected, due to extra CRDS parameters not having been "
         "implemented yet."
     )
-    RomanStep.from_cmdline(args)
+    with resource_tracker.track(log=request):
+        RomanStep.from_cmdline(args)
+
     rtdata.get_truth(f"truth/WFI/image/{output}")
 
     diff = compare_asdf(rtdata.output, rtdata.truth, **ignore_asdf_paths)
diff --git a/romancal/regtest/test_wfi_grism_16resultants.py b/romancal/regtest/test_wfi_grism_16resultants.py
index 06157dad9..7b4cc2c4c 100644
--- a/romancal/regtest/test_wfi_grism_16resultants.py
+++ b/romancal/regtest/test_wfi_grism_16resultants.py
@@ -10,7 +10,7 @@
 
 
 @pytest.fixture(scope="module")
-def run_elp(rtdata_module):
+def run_elp(rtdata_module, resource_tracker):
     rtdata = rtdata_module
 
     # The input data is from INS for stress testing at some point this should be generated
@@ -27,7 +27,8 @@ def run_elp(rtdata_module):
         "roman_elp",
         rtdata.input,
     ]
-    ExposurePipeline.from_cmdline(args)
+    with resource_tracker.track():
+        ExposurePipeline.from_cmdline(args)
     return rtdata
 
 
@@ -53,6 +54,10 @@ def input_model(input_filename):
         yield model
 
 
+def test_log_tracked_resources(log_tracked_resources, run_elp):
+    log_tracked_resources()
+
+
 def test_output_is_image_model(output_model):
     # DMS414
     assert isinstance(output_model, rdm.datamodels.ImageModel)
diff --git a/romancal/regtest/test_wfi_grism_pipeline.py b/romancal/regtest/test_wfi_grism_pipeline.py
index 16f4c6a62..50b310841 100644
--- a/romancal/regtest/test_wfi_grism_pipeline.py
+++ b/romancal/regtest/test_wfi_grism_pipeline.py
@@ -16,7 +16,7 @@
 
 
 @pytest.fixture(scope="module")
-def run_elp(rtdata_module):
+def run_elp(rtdata_module, resource_tracker):
     rtdata = rtdata_module
 
     input_data = "r0000201001001001001_0001_wfi01_uncal.asdf"
@@ -30,7 +30,8 @@ def run_elp(rtdata_module):
         "roman_elp",
         rtdata.input,
     ]
-    ExposurePipeline.from_cmdline(args)
+    with resource_tracker.track():
+        ExposurePipeline.from_cmdline(args)
     rtdata.get_truth(f"truth/WFI/grism/{output}")
     return rtdata
 
@@ -80,6 +81,10 @@ def repointed_filename_and_delta(output_filename):
     return repointed_filename, delta
 
 
+def test_log_tracked_resources(log_tracked_resources, run_elp):
+    log_tracked_resources()
+
+
 def test_output_matches_truth(output_filename, truth_filename, ignore_asdf_paths):
     diff = compare_asdf(output_filename, truth_filename, **ignore_asdf_paths)
     assert diff.identical, diff.report()
diff --git a/romancal/regtest/test_wfi_image_16resultants.py b/romancal/regtest/test_wfi_image_16resultants.py
index b4331cb78..a3fa4bb7b 100644
--- a/romancal/regtest/test_wfi_image_16resultants.py
+++ b/romancal/regtest/test_wfi_image_16resultants.py
@@ -10,7 +10,7 @@
 
 
 @pytest.fixture(scope="module")
-def run_elp(rtdata_module):
+def run_elp(rtdata_module, resource_tracker):
     rtdata = rtdata_module
 
     # The input data is from INS for stress testing at some point this should be generated
@@ -27,7 +27,8 @@ def run_elp(rtdata_module):
         "roman_elp",
         rtdata.input,
     ]
-    ExposurePipeline.from_cmdline(args)
+    with resource_tracker.track():
+        ExposurePipeline.from_cmdline(args)
     return rtdata
 
 
@@ -53,6 +54,10 @@ def input_model(input_filename):
         yield model
 
 
+def test_log_tracked_resources(log_tracked_resources, run_elp):
+    log_tracked_resources()
+
+
 def test_output_is_image_model(output_model):
     # DMS413
     assert isinstance(output_model, rdm.datamodels.ImageModel)
diff --git a/romancal/regtest/test_wfi_image_pipeline.py b/romancal/regtest/test_wfi_image_pipeline.py
index 0fd85a297..204554d95 100644
--- a/romancal/regtest/test_wfi_image_pipeline.py
+++ b/romancal/regtest/test_wfi_image_pipeline.py
@@ -18,7 +18,7 @@
 
 
 @pytest.fixture(scope="module")
-def run_elp(rtdata_module):
+def run_elp(rtdata_module, resource_tracker):
     rtdata = rtdata_module
 
     input_data = "r0000101001001001001_0001_wfi01_uncal.asdf"
@@ -32,7 +32,8 @@ def run_elp(rtdata_module):
         "roman_elp",
         rtdata.input,
     ]
-    ExposurePipeline.from_cmdline(args)
+    with resource_tracker.track():
+        ExposurePipeline.from_cmdline(args)
 
     # get truth file
     rtdata.get_truth(f"truth/WFI/image/{output}")
@@ -84,6 +85,10 @@ def repointed_filename_and_delta(output_filename):
     return repointed_filename, delta
 
 
+def test_log_tracked_resources(log_tracked_resources, run_elp):
+    log_tracked_resources()
+
+
 @pytest.mark.soctests
 def test_output_matches_truth(output_filename, truth_filename, ignore_asdf_paths):
     diff = compare_asdf(output_filename, truth_filename, **ignore_asdf_paths)
diff --git a/romancal/regtest/test_wfi_photom.py b/romancal/regtest/test_wfi_photom.py
index 7128f4cb3..04eb9695c 100644
--- a/romancal/regtest/test_wfi_photom.py
+++ b/romancal/regtest/test_wfi_photom.py
@@ -12,7 +12,9 @@
 
 
 @pytest.mark.bigdata
-def test_absolute_photometric_calibration(rtdata, ignore_asdf_paths):
+def test_absolute_photometric_calibration(
+    rtdata, ignore_asdf_paths, resource_tracker, request
+):
     """DMS140 Test: Testing application of photometric correction using
     CRDS selected photom file."""
 
@@ -47,7 +49,9 @@ def test_absolute_photometric_calibration(rtdata, ignore_asdf_paths):
         " The first ERROR is expected, due to extra CRDS parameters"
         " not having been implemented yet."
     )
-    RomanStep.from_cmdline(args)
+    with resource_tracker.track(log=request):
+        RomanStep.from_cmdline(args)
+
     photom_out = rdm.open(rtdata.output)
 
     step.log.info(
diff --git a/romancal/regtest/test_wfi_saturation.py b/romancal/regtest/test_wfi_saturation.py
index d71de4c1f..3e16f7359 100644
--- a/romancal/regtest/test_wfi_saturation.py
+++ b/romancal/regtest/test_wfi_saturation.py
@@ -12,7 +12,7 @@
 
 
 @pytest.mark.bigdata
-def test_saturation_image_step(rtdata, ignore_asdf_paths):
+def test_saturation_image_step(rtdata, ignore_asdf_paths, resource_tracker, request):
     """Testing retrieval of best ref file for image data,
     and creation of a ramp file with CRDS selected saturation file applied."""
 
@@ -34,7 +34,8 @@ def test_saturation_image_step(rtdata, ignore_asdf_paths):
     rtdata.output = output
 
     args = ["romancal.step.SaturationStep", rtdata.input]
-    RomanStep.from_cmdline(args)
+    with resource_tracker.track(log=request):
+        RomanStep.from_cmdline(args)
 
     ramp_out = rdm.open(rtdata.output)
     assert "roman.pixeldq" in ramp_out.to_flat_dict()
@@ -45,7 +46,7 @@ def test_saturation_image_step(rtdata, ignore_asdf_paths):
 
 
 @pytest.mark.bigdata
-def test_saturation_grism_step(rtdata, ignore_asdf_paths):
+def test_saturation_grism_step(rtdata, ignore_asdf_paths, resource_tracker, request):
     """Testing retrieval of best ref file for grism data,
     and creation of a ramp file with CRDS selected saturation file applied."""
 
@@ -67,7 +68,9 @@ def test_saturation_grism_step(rtdata, ignore_asdf_paths):
     rtdata.output = output
 
     args = ["romancal.step.SaturationStep", rtdata.input]
-    RomanStep.from_cmdline(args)
+    with resource_tracker.track(log=request):
+        RomanStep.from_cmdline(args)
+    resource_tracker.log(request)
 
     ramp_out = rdm.open(rtdata.output)
     assert "roman.pixeldq" in ramp_out.to_flat_dict()


Reviews:
user: login: zacharyburnett
body: suggest renaming "benchmark" to "usage tracking" 
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-24T20:11:26Z
user: login: zacharyburnett
body: typo
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-03-24T20:12:59Z
user: login: braingram
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-24T20:32:08Z
user: login: schlafly
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-26T13:35:13Z

Review comments:
diff_hunk: @@ -0,0 +1,52 @@
+import time
+import tracemalloc
+
+
+class TrackRuntime:
+    def __enter__(self):
+        self._t0 = time.monotonic()
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.value = time.monotonic() - self._t0
+
+    def log(self):
+        return ("bench-time", self.value)
path: romancal/regtest/benchmarks.py
user: login: zacharyburnett
body: ```suggestion
        return ("tracked-time", self.value)
```
created_at: 2025-03-24T19:56:02Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,52 @@
+import time
+import tracemalloc
+
+
+class TrackRuntime:
+    def __enter__(self):
+        self._t0 = time.monotonic()
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.value = time.monotonic() - self._t0
+
+    def log(self):
+        return ("bench-time", self.value)
+
+
+class TrackMemory:
+    def __enter__(self):
+        tracemalloc.start()
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        _, self.value = tracemalloc.get_traced_memory()
+        tracemalloc.stop()
+
+    def log(self):
+        return ("bench-peakmem", self.value)
path: romancal/regtest/benchmarks.py
user: login: zacharyburnett
body: ```suggestion
        return ("tracked-peakmem", self.value)
```
created_at: 2025-03-24T19:56:17Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,52 @@
+import time
+import tracemalloc
+
+
+class TrackRuntime:
+    def __enter__(self):
+        self._t0 = time.monotonic()
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.value = time.monotonic() - self._t0
+
+    def log(self):
+        return ("bench-time", self.value)
+
+
+class TrackMemory:
+    def __enter__(self):
+        tracemalloc.start()
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        _, self.value = tracemalloc.get_traced_memory()
+        tracemalloc.stop()
+
+    def log(self):
+        return ("bench-peakmem", self.value)
+
+
+class Benchmark:
path: romancal/regtest/benchmarks.py
user: login: zacharyburnett
body: ```suggestion
class Usage:
```
created_at: 2025-03-24T19:57:55Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,52 @@
+import time
+import tracemalloc
+
+
+class TrackRuntime:
+    def __enter__(self):
+        self._t0 = time.monotonic()
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.value = time.monotonic() - self._t0
+
+    def log(self):
+        return ("bench-time", self.value)
+
+
+class TrackMemory:
+    def __enter__(self):
+        tracemalloc.start()
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        _, self.value = tracemalloc.get_traced_memory()
+        tracemalloc.stop()
+
+    def log(self):
+        return ("bench-peakmem", self.value)
+
+
+class Benchmark:
+    def __init__(self):
+        self.trackers = [TrackMemory(), TrackRuntime()]
+
+    def __enter__(self):
+        [t.__enter__() for t in self.trackers]
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        [t.__exit__(exc_type, exc_value, traceback) for t in self.trackers]
+
+    def log(self, user_properties):
+        user_properties.extend(t.log() for t in self.trackers)
+
+
+class BenchmarkManager:
path: romancal/regtest/benchmarks.py
user: login: zacharyburnett
body: ```suggestion
class UsageTracker:
```
created_at: 2025-03-24T19:58:35Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,52 @@
+import time
+import tracemalloc
+
+
+class TrackRuntime:
+    def __enter__(self):
+        self._t0 = time.monotonic()
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.value = time.monotonic() - self._t0
+
+    def log(self):
+        return ("bench-time", self.value)
+
+
+class TrackMemory:
+    def __enter__(self):
+        tracemalloc.start()
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        _, self.value = tracemalloc.get_traced_memory()
+        tracemalloc.stop()
+
+    def log(self):
+        return ("bench-peakmem", self.value)
+
+
+class Benchmark:
+    def __init__(self):
+        self.trackers = [TrackMemory(), TrackRuntime()]
+
+    def __enter__(self):
+        [t.__enter__() for t in self.trackers]
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        [t.__exit__(exc_type, exc_value, traceback) for t in self.trackers]
+
+    def log(self, user_properties):
+        user_properties.extend(t.log() for t in self.trackers)
+
+
+class BenchmarkManager:
+    def __init__(self):
+        self.benchmarks = {}
path: romancal/regtest/benchmarks.py
user: login: zacharyburnett
body: ```suggestion
        self.usages = {}
```
created_at: 2025-03-24T19:58:54Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,52 @@
+import time
+import tracemalloc
+
+
+class TrackRuntime:
+    def __enter__(self):
+        self._t0 = time.monotonic()
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.value = time.monotonic() - self._t0
+
+    def log(self):
+        return ("bench-time", self.value)
+
+
+class TrackMemory:
+    def __enter__(self):
+        tracemalloc.start()
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        _, self.value = tracemalloc.get_traced_memory()
+        tracemalloc.stop()
+
+    def log(self):
+        return ("bench-peakmem", self.value)
+
+
+class Benchmark:
+    def __init__(self):
+        self.trackers = [TrackMemory(), TrackRuntime()]
+
+    def __enter__(self):
+        [t.__enter__() for t in self.trackers]
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        [t.__exit__(exc_type, exc_value, traceback) for t in self.trackers]
+
+    def log(self, user_properties):
+        user_properties.extend(t.log() for t in self.trackers)
+
+
+class BenchmarkManager:
+    def __init__(self):
+        self.benchmarks = {}
+
+    def __call__(self, name):
+        benchmark = Benchmark()
+        self.benchmarks[name] = benchmark
+        return benchmark
path: romancal/regtest/benchmarks.py
user: login: zacharyburnett
body: ```suggestion
    def track(self, name):
        usage = Usage()
        self.usages[name] = usage
        return usage
```
created_at: 2025-03-24T19:59:52Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,52 @@
+import time
+import tracemalloc
+
+
+class TrackRuntime:
+    def __enter__(self):
+        self._t0 = time.monotonic()
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.value = time.monotonic() - self._t0
+
+    def log(self):
+        return ("bench-time", self.value)
+
+
+class TrackMemory:
+    def __enter__(self):
+        tracemalloc.start()
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        _, self.value = tracemalloc.get_traced_memory()
+        tracemalloc.stop()
+
+    def log(self):
+        return ("bench-peakmem", self.value)
+
+
+class Benchmark:
+    def __init__(self):
+        self.trackers = [TrackMemory(), TrackRuntime()]
+
+    def __enter__(self):
+        [t.__enter__() for t in self.trackers]
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        [t.__exit__(exc_type, exc_value, traceback) for t in self.trackers]
+
+    def log(self, user_properties):
+        user_properties.extend(t.log() for t in self.trackers)
+
+
+class BenchmarkManager:
+    def __init__(self):
+        self.benchmarks = {}
+
+    def __call__(self, name):
+        benchmark = Benchmark()
+        self.benchmarks[name] = benchmark
+        return benchmark
+
+    def log(self, name, user_properties):
+        self.benchmarks[name].log(user_properties)
path: romancal/regtest/benchmarks.py
user: login: zacharyburnett
body: ```suggestion
    def log(self, name, user_properties):
        self.usages[name].log(user_properties)
```
created_at: 2025-03-24T20:01:16Z
author_association: COLLABORATOR
diff_hunk: @@ -9,6 +9,7 @@
 from ci_watson.artifactory_helpers import UPLOAD_SCHEMA
 from numpy.testing import assert_allclose, assert_equal
 
+from romancal.regtest.benchmarks import BenchmarkManager
path: romancal/regtest/conftest.py
user: login: zacharyburnett
body: ```suggestion
from romancal.regtest.usages import UsageTracker
```
created_at: 2025-03-24T20:01:34Z
author_association: COLLABORATOR
diff_hunk: @@ -233,6 +234,19 @@ def ignore_asdf_paths():
     return {"ignore": ignore_attr}
 
 
+@pytest.fixture(scope="module")
+def benchmark():
+    return BenchmarkManager()
path: romancal/regtest/conftest.py
user: login: zacharyburnett
body: ```suggestion
def track_usage():
    return UsageTracker()
```
created_at: 2025-03-24T20:02:03Z
author_association: COLLABORATOR
diff_hunk: @@ -233,6 +234,19 @@ def ignore_asdf_paths():
     return {"ignore": ignore_attr}
 
 
+@pytest.fixture(scope="module")
+def benchmark():
+    return BenchmarkManager()
+
+
+@pytest.fixture()
+def log_benchmark(benchmark, request):
+    def callback(fixture_name):
+        benchmark.log(fixture_name, request.node.user_properties)
path: romancal/regtest/conftest.py
user: login: zacharyburnett
body: ```suggestion
def log_usage(track_usage, request):
    def callback(fixture_name):
        track_usage.log(fixture_name, request.node.user_properties)
```
created_at: 2025-03-24T20:02:34Z
author_association: COLLABORATOR
diff_hunk: @@ -11,12 +11,14 @@
 from . import util
 from .regtestdata import compare_asdf
 
+BENCHMARK_NAME = "mosiac_pipeline"
+
 # mark all tests in this module
 pytestmark = [pytest.mark.bigdata, pytest.mark.soctests]
 
 
 @pytest.fixture(scope="module")
-def run_mos(rtdata_module):
+def run_mos(rtdata_module, benchmark):
path: romancal/regtest/test_mos_pipeline.py
user: login: zacharyburnett
body: ```suggestion
def run_mos(rtdata_module, track_usage):
```
created_at: 2025-03-24T20:04:11Z
author_association: COLLABORATOR
diff_hunk: @@ -29,7 +31,8 @@ def run_mos(rtdata_module):
         "roman_mos",
         rtdata.input,
     ]
-    MosaicPipeline.from_cmdline(args)
+    with benchmark(BENCHMARK_NAME):
path: romancal/regtest/test_mos_pipeline.py
user: login: zacharyburnett
body: ```suggestion
    with track_usage(USAGE_LOG_NAME):
```
created_at: 2025-03-24T20:04:36Z
author_association: COLLABORATOR
diff_hunk: @@ -67,6 +70,10 @@ def preview_filename(output_filename):
     return preview_filename
 
 
+def test_benchmark(log_benchmark, run_mos):
+    log_benchmark(BENCHMARK_NAME)
path: romancal/regtest/test_mos_pipeline.py
user: login: zacharyburnett
body: ```suggestion
def test_benchmark(log_usage, run_mos):
    log_usage(USAGE_LOG_NAME)
```
created_at: 2025-03-24T20:05:13Z
author_association: COLLABORATOR
diff_hunk: @@ -7,12 +7,14 @@
 from . import util
 from .regtestdata import compare_asdf
 
+BENCHMARK_NAME = "mosiac_pipeline_skycell"
+
 # mark all tests in this module
 pytestmark = [pytest.mark.bigdata, pytest.mark.soctests]
 
 
 @pytest.fixture(scope="module")
-def run_mos(rtdata_module):
+def run_mos(rtdata_module, benchmark):
path: romancal/regtest/test_mos_skycell_pipeline.py
user: login: zacharyburnett
body: ```suggestion
def run_mos(rtdata_module, track_usage):
```
created_at: 2025-03-24T20:06:07Z
author_association: COLLABORATOR
diff_hunk: @@ -23,7 +25,8 @@ def run_mos(rtdata_module):
         "roman_mos",
         rtdata.input,
     ]
-    MosaicPipeline.from_cmdline(args)
+    with benchmark(BENCHMARK_NAME):
path: romancal/regtest/test_mos_skycell_pipeline.py
user: login: zacharyburnett
body: ```suggestion
    with track_usage(USAGE_LOG_NAME):
```
created_at: 2025-03-24T20:06:29Z
author_association: COLLABORATOR
diff_hunk: @@ -44,6 +47,10 @@ def truth_filename(run_mos):
     return run_mos.truth
 
 
+def test_benchmark(log_benchmark, run_mos):
+    log_benchmark(BENCHMARK_NAME)
path: romancal/regtest/test_mos_skycell_pipeline.py
user: login: zacharyburnett
body: ```suggestion
def test_log_usage(log_usage, run_mos):
    log_usage(USAGE_LOG_NAME)
```
created_at: 2025-03-24T20:07:21Z
author_association: COLLABORATOR
diff_hunk: @@ -11,12 +11,14 @@
 from . import util
 from .regtestdata import compare_asdf
 
+BENCHMARK_NAME = "mosiac_pipeline"
path: romancal/regtest/test_mos_pipeline.py
user: login: zacharyburnett
body: ```suggestion
USAGE_LOG_NAME = "mosiac_pipeline"
```
created_at: 2025-03-24T20:09:56Z
author_association: COLLABORATOR
diff_hunk: @@ -11,12 +11,14 @@
 from . import util
 from .regtestdata import compare_asdf
 
+BENCHMARK_NAME = "mosiac_pipeline"
path: romancal/regtest/test_mos_pipeline.py
user: login: zacharyburnett
body: ```suggestion
BENCHMARK_NAME = "mosaic_pipeline"
```
created_at: 2025-03-24T20:12:07Z
author_association: COLLABORATOR
diff_hunk: @@ -7,12 +7,14 @@
 from . import util
 from .regtestdata import compare_asdf
 
+BENCHMARK_NAME = "mosiac_pipeline_skycell"
path: romancal/regtest/test_mos_skycell_pipeline.py
user: login: zacharyburnett
body: ```suggestion
USAGE_LOG_NAME = "mosaic_pipeline_skycell"
```
created_at: 2025-03-24T20:12:36Z
author_association: COLLABORATOR
diff_hunk: @@ -7,12 +7,14 @@
 from . import util
 from .regtestdata import compare_asdf
 
+BENCHMARK_NAME = "mosiac_pipeline_skycell"
path: romancal/regtest/test_mos_skycell_pipeline.py
user: login: braingram
body: Thanks! Fixed the typo in https://github.com/spacetelescope/romancal/pull/1664/commits/193a09fff30611f29d0c6b38d9ab959406cd2bc9
created_at: 2025-03-24T20:32:08Z
author_association: COLLABORATOR

PR comments:
user: login: codecov[bot]
created_at: 2025-03-24T19:52:13Z
body: ## [Codecov](https://app.codecov.io/gh/spacetelescope/romancal/pull/1664?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 79.91%. Comparing base [(`8ab6091`)](https://app.codecov.io/gh/spacetelescope/romancal/commit/8ab60914019fb3344aa2c2fdf96cf16bcd979b77?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) to head [(`926d9b9`)](https://app.codecov.io/gh/spacetelescope/romancal/commit/926d9b91924ef2ec7e6b1a0cebbe7bb73e357d04?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope).
> Report is 149 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #1664      +/-   ##
==========================================
+ Coverage   79.81%   79.91%   +0.09%     
==========================================
  Files         115      116       +1     
  Lines        6546     6577      +31     
==========================================
+ Hits         5225     5256      +31     
  Misses       1321     1321              
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/spacetelescope/romancal/pull/1664?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: braingram
created_at: 2025-03-24T20:31:54Z
body: > suggest renaming "benchmark" to "usage tracking"

Thanks! I made the rename in https://github.com/spacetelescope/romancal/pull/1664/commits/193a09fff30611f29d0c6b38d9ab959406cd2bc9 although I went with your earlier suggestion (made offline) of `ResourceTracker` instead of `UsageTracker`. I'll update the description with the new API and property names ("tracked-time/peakmem" instead of "bench-time/peakmen")
author_association: COLLABORATOR
user: login: schlafly
created_at: 2025-03-25T14:15:38Z
body: Cool!  I agree it's smart to take advantage of the regtests for this.
author_association: COLLABORATOR
user: login: schlafly
created_at: 2025-03-25T19:31:11Z
body: This code looks good to me.  Regtests pass.  I don't think we need more docs for this kind of internal testing thing, though one could conceivably imagine a page here https://github.com/spacetelescope/romancal/wiki , though probably the module level docstrings in resource_tracker are already good.  Anything else you want me to do before I merge this?  Do I understand correctly that while this constructs an xml report at present, that isn't yet hooked in to any more visual display?
author_association: COLLABORATOR
user: login: braingram
created_at: 2025-03-25T19:38:06Z
body: > This code looks good to me. Regtests pass. I don't think we need more docs for this kind of internal testing thing, though one could conceivably imagine a page here https://github.com/spacetelescope/romancal/wiki , though probably the module level docstrings in resource_tracker are already good. Anything else you want me to do before I merge this? Do I understand correctly that while this constructs an xml report at present, that isn't yet hooked in to any more visual display?

Thanks! One question for you is if there are other regtests that would be worth tracking at this point? I tried to go through and add hooks to all the regtests that call a pipeline/step (although I may have missed some).

Let me know if more docs are helpful. At some point this will likely be moved to ci_watson or some other shared location to also be used by jwst. Once it's moved a RTD page seems like a good fit. Especially something that highlights how to read results etc.

For viewing the results one plan is to finalize https://github.com/spacetelescope/RegressionTests/pull/164 to use pytestresultsdiff to add a performance report for PR related regtest runs. Otherwise the tooling isn't anything that I'm aware of and certainly something to flush out once we get a feel for how reliable and useful these measurements are.
author_association: COLLABORATOR
user: login: braingram
created_at: 2025-03-25T20:05:00Z
body: FYI From offline conversation with @tapastro he's interested in using this with jwst sooner rather than later so I'm gonna take a look at moving the fixtures to a shared library. I think we have 2 options with this PR:
- delay it until the fixtures are in the shared library to use
- move forward with this PR knowing that a follow up can be made once the fixtures are in the shared library

There are no plans to change the fixtures so I'm in favor of option 2 (move forward with this PR then clean things up once the shared version is available) but either is fine with me. 
author_association: COLLABORATOR
user: login: schlafly
created_at: 2025-03-25T20:15:16Z
body: I agree that step times are the right thing to do.  Probably the exposure and mosaic pipeline runs that hit all the individual steps are the things that it would be most easy to actually stay on top of, with the individual step-level tests more valuable for drilling down when one notices that something significant happened with the exposure or mosaic pipelines.

Practically I think this is more benchmarks than we need; i.e., in a vacuum I would not have separately benchmarked the normal ELP run and the "repointed" ELP run.  But since we're running both there's no reason not to compute the processing & memory statistics.

Basically I think you hit the right tests.  I think the docs look good.  Moving to a shared Webb / Roman location makes sense to me but I'll follow your advice on when to merge; happy to go with your option 2.
author_association: COLLABORATOR