Repository: spacetelescope/jwst
Pull request #9073: JP-3790 add residual fringe correction to extract1d table
Author: login: jemorrison
Labels: name: extract_1d
name: MIRI
name: spec3 pipeline
name: documentation
name: regression_testing
Created: 2025-01-13T22:15:56Z | Merged: 2025-03-07T16:17:21Z

<!-- If this PR closes a JIRA ticket, make sure the title starts with the JIRA issue number,
for example JP-1234: <Fix a bug> -->
Resolves [JP-3790](https://jira.stsci.edu/browse/JP-3790)

<!-- If this PR closes a GitHub issue, reference it here by its number -->
Closes #8920 

<!-- describe the changes comprising this PR here -->
This PR adds three columns to the extracted 1d spectra for MIRI MRS data. These three columns are related to residual fringe correction. It is assumed that a new parameter pars reference file will be delivered for MIRI MRS spec3 which will set ifu_rfcorr = True. If the IFU cube that the spectra is being extracted from is for a single band then the residual fringe corrected data is reported as 3 additional columns. If the data is not for a single band then Nan values are written to these three additional columns. For default spec2 extracted spectrum the data does not cover 1 band (but it is for 2 channels) so the residual fringe corrected data will not be written to the x1d files - Nan values will be written. 


The spectral leak step was updated to accept MRSMultiSpecModel. 

The spectral leak has no unit tests. A separate JP ticket will handle adding various unit tests for this step. 
https://jira.stsci.edu/browse/JP-3911

Other steps that use MRS x1d data was reviewed to see if it need to be updated to accept MRSMultSpecModel data.
The master background step and combine 1d step use datamodels.open to open the X1d files. The Flux column is read from the x1d files. No change in these steps is required.

This PR also updated the moving target retests that had an error in them. This why there are regression test failures with this mode

<!-- if you can't perform these tasks due to permissions, please ask a maintainer to do them -->
## Tasks
- [x] **request a review from someone specific**, to avoid making the maintainers review every PR
- [x] add a build milestone, i.e. `Build 11.3` (use the [latest build](https://github.com/spacetelescope/jwst/milestones) if not sure)
- [x] Does this PR change user-facing code / API? (if not, label with `no-changelog-entry-needed`)
  - [x] write news fragment(s) in `changes/`: `echo "changed something" > changes/<PR#>.<changetype>.rst` (see below for change types) 
  - [x] update or add relevant tests
  - [x] update relevant docstrings and / or `docs/` page
  - [x] [start a regression test](https://github.com/spacetelescope/RegressionTests/actions/workflows/jwst.yml) and include a link to the running job ([click here for instructions]
  https://github.com/spacetelescope/RegressionTests/actions/runs/13664659037
  - [ ] (https://github.com/spacetelescope/RegressionTests/blob/main/docs/running_regression_tests.md))
    - [x] Do truth files need to be updated ("okified")?
      - [ ] **after the reviewer has approved these changes**, run `okify_regtests` to update the truth files
- [ ] if a JIRA ticket exists, [make sure it is resolved properly](https://github.com/spacetelescope/jwst/wiki/How-to-resolve-JIRA-issues)

<details><summary>news fragment change types...</summary>

- ``changes/<PR#>.general.rst``: infrastructure or miscellaneous change
- ``changes/<PR#>.docs.rst``
- ``changes/<PR#>.stpipe.rst``
- ``changes/<PR#>.datamodels.rst``
- ``changes/<PR#>.scripts.rst``
- ``changes/<PR#>.fits_generator.rst``
- ``changes/<PR#>.set_telescope_pointing.rst``
- ``changes/<PR#>.pipeline.rst``

## stage 1
- ``changes/<PR#>.group_scale.rst``
- ``changes/<PR#>.dq_init.rst``
- ``changes/<PR#>.emicorr.rst``
- ``changes/<PR#>.saturation.rst``
- ``changes/<PR#>.ipc.rst``
- ``changes/<PR#>.firstframe.rst``
- ``changes/<PR#>.lastframe.rst``
- ``changes/<PR#>.reset.rst``
- ``changes/<PR#>.superbias.rst``
- ``changes/<PR#>.refpix.rst``
- ``changes/<PR#>.linearity.rst``
- ``changes/<PR#>.rscd.rst``
- ``changes/<PR#>.persistence.rst``
- ``changes/<PR#>.dark_current.rst``
- ``changes/<PR#>.charge_migration.rst``
- ``changes/<PR#>.jump.rst``
- ``changes/<PR#>.clean_flicker_noise.rst``
- ``changes/<PR#>.ramp_fitting.rst``
- ``changes/<PR#>.gain_scale.rst``

## stage 2
- ``changes/<PR#>.assign_wcs.rst``
- ``changes/<PR#>.badpix_selfcal.rst``
- ``changes/<PR#>.msaflagopen.rst``
- ``changes/<PR#>.nsclean.rst``
- ``changes/<PR#>.imprint.rst``
- ``changes/<PR#>.background.rst``
- ``changes/<PR#>.extract_2d.rst``
- ``changes/<PR#>.master_background.rst``
- ``changes/<PR#>.wavecorr.rst``
- ``changes/<PR#>.srctype.rst``
- ``changes/<PR#>.straylight.rst``
- ``changes/<PR#>.wfss_contam.rst``
- ``changes/<PR#>.flatfield.rst``
- ``changes/<PR#>.fringe.rst``
- ``changes/<PR#>.pathloss.rst``
- ``changes/<PR#>.barshadow.rst``
- ``changes/<PR#>.photom.rst``
- ``changes/<PR#>.pixel_replace.rst``
- ``changes/<PR#>.resample_spec.rst``
- ``changes/<PR#>.residual_fringe.rst``
- ``changes/<PR#>.cube_build.rst``
- ``changes/<PR#>.extract_1d.rst``
- ``changes/<PR#>.resample.rst``

## stage 3
- ``changes/<PR#>.assign_mtwcs.rst``
- ``changes/<PR#>.mrs_imatch.rst``
- ``changes/<PR#>.tweakreg.rst``
- ``changes/<PR#>.skymatch.rst``
- ``changes/<PR#>.exp_to_source.rst``
- ``changes/<PR#>.outlier_detection.rst``
- ``changes/<PR#>.tso_photometry.rst``
- ``changes/<PR#>.stack_refs.rst``
- ``changes/<PR#>.align_refs.rst``
- ``changes/<PR#>.klip.rst``
- ``changes/<PR#>.spectral_leak.rst``
- ``changes/<PR#>.source_catalog.rst``
- ``changes/<PR#>.combine_1d.rst``
- ``changes/<PR#>.ami.rst``

## other
- ``changes/<PR#>.wfs_combine.rst``
- ``changes/<PR#>.white_light.rst``
- ``changes/<PR#>.cube_skymatch.rst``
- ``changes/<PR#>.engdb_tools.rst``
- ``changes/<PR#>.guider_cds.rst``
</details>


Linked issue #8920: Revise MRS 1d residual fringe correction to run automatically and populate x1d extension
_Issue [JP-3790](https://jira.stsci.edu/browse/JP-3790) was created on JIRA by [David Law](https://jira.stsci.edu/secure/ViewProfile.jspa?name=dlaw):_

After considerable discussion within the MIRI team, we would like to revise the current implementation of the 1d residual fringe correction within the extract_1d step.

At present, ifu_rfcorr is an optional argument that users can provide to apply the 1d correction.  This correction is off by default as it can remove periodic features in some cases with molecular bands, but this also means that the vast majority of users need to rerun both cube building (to create per-band cubes) and spectral extraction in order to get clean spectra.

The requested change is to have 1d residual fringe correction always run by default, but to provide two versions of the extracted spectra.  The first copy, in the usual EXTRACT1D 1st extension, will be the normal spectrum with no defringing applied.  The defringed spectra (science, background, etc) will be in a new 2nd extension with some sensible FITS EXTNAME.  Maybe RFCORR1D?

This way, users can simply choose for themselves which to use without rerunning the pipeline, or requiring additional MAST data products.

A few details:
 * It may be worth retaining the ifu_rfcorr option flag in extract_1d for a little while, but when set to TRUE simply log a statement informing users of the change in behaviour.  This at least won't break previous code.
 * At the same time, the MIRI team will deliver a parameter reference file update that changes the default Spec3 cubes to use per-band cubes for which residual fringe correction works well.  This will have a significant impact on DMS and MAST.
 * We'll presumably need an updated extract1d datamodel to handle the new extension?  Not sure of the details here.
Linked issue comments: user: login: stscijgbot-jp
created_at: 2025-01-07T17:23:05Z
body: _Comment by [Jane Morrison](https://jira.stsci.edu/secure/ViewProfile.jspa?name=morrison) on [JIRA](https://jira.stsci.edu/browse/JP-3790?focusedCommentId=938554#comment-938554):_

[David Law](https://jira.stsci.edu/secure/ViewProfile.jspa?name=dlaw) 
Here is what I am thinking on the logic.
Let ifu_rfcorr = False by default. If it is false then run extract 1d twice. First time no residual fringe correction, second time residual fringe correction.
I will update the datamodel to extract1d that will allow two spec_tables in output.
If the user has set ifu_rfcorr = True then they will expect the first extension to contain the residual fringe corrected data so only run extract1d once and  it will apply the residual fringe correction. The output will only have 1 spec_table 



Does that make sense  to you ? I was trying to keep it working the way it does now if the user has set ifu_rfcorr to True.
Of course I will need to make the documentation clear on what is going on.
author_association: COLLABORATOR
user: login: stscijgbot-jp
created_at: 2025-01-09T00:13:04Z
body: _Comment by [Jane Morrison](https://jira.stsci.edu/secure/ViewProfile.jspa?name=morrison) on [JIRA](https://jira.stsci.edu/browse/JP-3790?focusedCommentId=939007#comment-939007):_

[Melanie Clarke](https://jira.stsci.edu/secure/ViewProfile.jspa?name=mclarke) [Tyler Pauly](https://jira.stsci.edu/secure/ViewProfile.jspa?name=tpauly) after looking over the JP-3794: Add trace-based extraction option for NIRSpec and how the 'use_source_position' value is set None - I wonder if I could do something similar. 
Default if_rfcorr = None. If none then first run extract 1d with no rf correction and then run it a second time with rf correction. The output extract1d file will have 2 spec tables.
If ifu_rfcorr= True (user set) then only run extract 1d once and the output extract1d only has the rf spec table.
if ifu_rfcorr=False (user set) then only run extract 1d once and the output extract1d only has spec table without rf correction.
In both cases (True and False) a log message could be written to screen telling user of the None option. 



 
author_association: COLLABORATOR
user: login: stscijgbot-jp
created_at: 2025-01-09T16:28:09Z
body: _Comment by [Melanie Clarke](https://jira.stsci.edu/secure/ViewProfile.jspa?name=mclarke) on [JIRA](https://jira.stsci.edu/browse/JP-3790?focusedCommentId=939099#comment-939099):_

[Jane Morrison](https://jira.stsci.edu/secure/ViewProfile.jspa?name=morrison) - rereading the ticket here and catching up, it sounds to me like what is requested is that we always run fringe correction and always provide two versions of the spectra.  The ifu_rfcorr parameter can be maintained for a while, but it will have no effect: we don't want the user to have control over this behavior, we want all data products to be consistent.  

[David Law](https://jira.stsci.edu/secure/ViewProfile.jspa?name=dlaw) - can you confirm this interpretation?
author_association: COLLABORATOR
user: login: stscijgbot-jp
created_at: 2025-01-09T19:53:05Z
body: _Comment by [Jane Morrison](https://jira.stsci.edu/secure/ViewProfile.jspa?name=morrison) on [JIRA](https://jira.stsci.edu/browse/JP-3790?focusedCommentId=939171#comment-939171):_

[David Law](https://jira.stsci.edu/secure/ViewProfile.jspa?name=dlaw) 
Melanie and I were discussing the cleanest way to add residual fringe information in the extracted data. Instead of adding an additional extension to the output data we were thinking we could instead add 3 columns to the extracted spec table that hold the residual fringe corrected flux, surface brightness and background. Users will need to update their code to read in these columns but they would need to update their code to read in the residual fringe corrected data anyway (which would be the second extension). Is adding these 3 columns acceptable to you instead of having two extension ?
author_association: COLLABORATOR
user: login: stscijgbot-jp
created_at: 2025-01-09T20:03:06Z
body: _Comment by [Tyler Pauly](https://jira.stsci.edu/secure/ViewProfile.jspa?name=tpauly) on [JIRA](https://jira.stsci.edu/browse/JP-3790?focusedCommentId=939176#comment-939176):_

Would this use a new datamodel to capture the additional columns, or would this change the table structure for all extract_1d products?
author_association: COLLABORATOR
user: login: stscijgbot-jp
created_at: 2025-01-09T20:33:05Z
body: _Comment by [Melanie Clarke](https://jira.stsci.edu/secure/ViewProfile.jspa?name=mclarke) on [JIRA](https://jira.stsci.edu/browse/JP-3790?focusedCommentId=939186#comment-939186):_

We were thinking a new datamodel - MRSSpecTable, maybe, since it's unlikely this would be needed for any other modes.
author_association: COLLABORATOR
user: login: stscijgbot-jp
created_at: 2025-01-10T16:48:11Z
body: _Comment by [David Law](https://jira.stsci.edu/secure/ViewProfile.jspa?name=dlaw) on [JIRA](https://jira.stsci.edu/browse/JP-3790?focusedCommentId=939298#comment-939298):_

[Jane Morrison](https://jira.stsci.edu/secure/ViewProfile.jspa?name=morrison) [Melanie Clarke](https://jira.stsci.edu/secure/ViewProfile.jspa?name=mclarke) That's right; for consistency we want to always do the same thing, providing two versions of the spectra.  The ifu_rfcorr parameter would ultimately go away, and simply be kept for a couple of builds to avoid crashing the pipeline if specified (instead printing out a warning message informing users of the new behaviour).  That does mean folks will need to get used to looking for the new values, but I think is probably less confusing in the long run.

Regarding adding columns instead of an extension, I think that sounds like a good modification.

A few additional clarifications based on the above:
 * The master_background step shouldn't need to be changed.  It uses x1d files from spec2, but we WANT to keep using the normal spectra without 1d fringe correction to subtract from the data cubes.
 * The spectral leak correction step will also need to be updated.  Currently it uses the FLUX extension of 1B to correct the FLUX extension of 3A, but we'll want to add using FLUX_RFCORR1D (or whatever we call it) from 1B to correct FLUX_RFCORR1D of 3A.
author_association: COLLABORATOR
user: login: stscijgbot-jp
created_at: 2025-01-13T22:18:06Z
body: _Comment by [Jane Morrison](https://jira.stsci.edu/secure/ViewProfile.jspa?name=morrison) on [JIRA](https://jira.stsci.edu/browse/JP-3790?focusedCommentId=939738#comment-939738):_

A few follow up questions. I pushed what I have so far to #9073 (so you can see how I am working this) 

[David Law](https://jira.stsci.edu/secure/ViewProfile.jspa?name=dlaw) The 1-D residual fringe correction is best applied to "band" cubes. What about the output from Spec2 ? To see what would happen I ran extract 1d with ifu_rfcorr=True on cubes from spec2 and it failed.  So in that case do we want to keep the ifu_rfcorr = False ?
Also what should I do about a case where the ifucube is not a band cube. Say someone creates a channel cube - or even a multichannel cube  and then tries to run the extract1d on that data ? 

[Melanie Clarke](https://jira.stsci.edu/secure/ViewProfile.jspa?name=mclarke) I created a new datamodel (see <https://github.com/spacetelescope/stdatamodels/pull/376])> which creates a mrs_spectable. I need to resolve the problem that currently the ifu.py  ifu_extract1d assumes a multi spec model. Either I need update the multi spec model to accept an mrs_spectable or I need to update the mrs_spec.schema to hold the the basic slit information contained in multi spec.schema.yaml (slit name, slit_ra...) 
Do you have a preference ? I don't think ifu data will have multi spec tables so we could put the needed info in mrs_spec.schema. 
author_association: COLLABORATOR
user: login: stscijgbot-jp
created_at: 2025-01-14T16:43:07Z
body: _Comment by [David Law](https://jira.stsci.edu/secure/ViewProfile.jspa?name=dlaw) on [JIRA](https://jira.stsci.edu/browse/JP-3790?focusedCommentId=939830#comment-939830):_

Hm, certainly I wouldn't expect it to get a meaningful correction for anything other than band cubes, though interesting that it actually crashes.

I'd be inclined to say that we don't even try running the rf code unless in from 'band' type cubes.  It should be possible to detect this looking at the CHANNEL and BAND header keywords; require CHANNEL='1', '2', '3', or '4' and BAND='SHORT', 'MEDIUM', or 'LONG'.  Anything with CHANNEL='12' or '34' or BAND='MULTIPLE' shouldn't be rf corrected, and in this case the pipeline should log an info statement about how RF correction won't be applied.

That in turn begs the question of what to do with the new rf_ columns in the Extract1d output if the rf correction hasn't been run.  We could duplicate the regular flux/surf_bright/etc extensions but that seems potentially misleading, and it may be better for them to be all-NaN values to make clear that the correction didn't get run.

'.
author_association: COLLABORATOR
user: login: stscijgbot-jp
created_at: 2025-01-14T16:43:07Z
body: _Comment by [Melanie Clarke](https://jira.stsci.edu/secure/ViewProfile.jspa?name=mclarke) on [JIRA](https://jira.stsci.edu/browse/JP-3790?focusedCommentId=939832#comment-939832):_

Jane and I discussed this offline, but for anyone else following along, I'll look into the datamodels issue myself, for my own edification.  I'd like to experiment with a couple options for how to structure the schema.
author_association: COLLABORATOR
user: login: jemorrison
created_at: 2025-01-14T16:53:45Z
body: I think if the cubes are not 'band' cubes then it might be best to set the 3 new columns to N/A so it clear the 1D residual fringe correction did not run 
author_association: COLLABORATOR
user: login: stscijgbot-jp
created_at: 2025-01-16T13:33:07Z
body: _Comment by [Melanie Clarke](https://jira.stsci.edu/secure/ViewProfile.jspa?name=mclarke) on [JIRA](https://jira.stsci.edu/browse/JP-3790?focusedCommentId=940220#comment-940220):_

Datamodels support is in this PR: <https://github.com/spacetelescope/stdatamodels/pull/377],> now merged.

Assigning back to Jane for the jwst work.
author_association: COLLABORATOR
user: login: stscijgbot-jp
created_at: 2025-03-07T16:18:06Z
body: _Comment by [Melanie Clarke](https://jira.stsci.edu/secure/ViewProfile.jspa?name=mclarke) on [JIRA](https://jira.stsci.edu/browse/JP-3790?focusedCommentId=948833#comment-948833):_

Fixed by #9073
author_association: COLLABORATOR
user: login: stscijgbot-jp
created_at: 2025-03-07T20:43:07Z
body: _Comment by [David Law](https://jira.stsci.edu/secure/ViewProfile.jspa?name=dlaw) on [JIRA](https://jira.stsci.edu/browse/JP-3790?focusedCommentId=948928#comment-948928):_

Hm, I just tried testing this and it looks like there's a problem somewhere.  Here's a plot of the regular and rf-corrected spectrum when I try running this on one example point source.  There shouldn't be a flux offset between the two.

Will try to look at the code and see if I can see where the issue is occurring.  [Jane Morrison](https://jira.stsci.edu/secure/ViewProfile.jspa?name=morrison) 

!test_mar7.png!
author_association: COLLABORATOR
user: login: stscijgbot-jp
created_at: 2025-03-07T20:53:08Z
body: _Comment by [David Law](https://jira.stsci.edu/secure/ViewProfile.jspa?name=dlaw) on [JIRA](https://jira.stsci.edu/browse/JP-3790?focusedCommentId=948931#comment-948931):_

I think I see the issue; the aperture correction is happening after creation of the rf-corrected versions, and is not being applied to the rf-corrected spectra.
author_association: COLLABORATOR
user: login: jemorrison
created_at: 2025-03-07T22:07:48Z
body: Ok I did will see how to fix this.
@melanieclarke should I start a new PR for this work  or do we pull it out of Main and fix it in the old PR. 

author_association: COLLABORATOR
user: login: stscijgbot-jp
created_at: 2025-03-10T13:38:06Z
body: _Comment by [Melanie Clarke](https://jira.stsci.edu/secure/ViewProfile.jspa?name=mclarke) on [JIRA](https://jira.stsci.edu/browse/JP-3790?focusedCommentId=949060#comment-949060):_

[Jane Morrison](https://jira.stsci.edu/secure/ViewProfile.jspa?name=morrison) - let's make a new PR for follow up.
author_association: COLLABORATOR
user: login: stscijgbot-jp
created_at: 2025-03-10T21:23:06Z
body: _Comment by [Melanie Clarke](https://jira.stsci.edu/secure/ViewProfile.jspa?name=mclarke) on [JIRA](https://jira.stsci.edu/browse/JP-3790?focusedCommentId=949271#comment-949271):_

Aperture correction applied in a follow-up PR:

#9260
author_association: COLLABORATOR
user: login: stscijgbot-jp
created_at: 2025-03-10T21:58:08Z
body: _Comment by [David Law](https://jira.stsci.edu/secure/ViewProfile.jspa?name=dlaw) on [JIRA](https://jira.stsci.edu/browse/JP-3790?focusedCommentId=949277#comment-949277):_

LG2M!  Closing.
author_association: COLLABORATOR

Diff:
diff --git a/changes/9073.extract_1d.rst b/changes/9073.extract_1d.rst
new file mode 100644
index 0000000000..77792baae7
--- /dev/null
+++ b/changes/9073.extract_1d.rst
@@ -0,0 +1 @@
+For MIRI MRS data added residual fringe corrected columns to extracted spectrum table
diff --git a/docs/jwst/extract_1d/description.rst b/docs/jwst/extract_1d/description.rst
index fd967e9298..b1b2861261 100644
--- a/docs/jwst/extract_1d/description.rst
+++ b/docs/jwst/extract_1d/description.rst
@@ -77,7 +77,11 @@ be an output table extension with the name EXTRACT1D.  This extension will
 have columns WAVELENGTH, FLUX, FLUX_ERROR, FLUX_VAR_POISSON, FLUX_VAR_RNOISE,
 FLUX_VAR_FLAT, SURF_BRIGHT, SB_ERROR, SB_VAR_POISSON, SB_VAR_RNOISE,
 SB_VAR_FLAT, DQ, BACKGROUND, BKGD_ERROR, BKGD_VAR_POISSON, BKGD_VAR_RNOISE,
-BKGD_VAR_FLAT and NPIXELS. Some metadata for the slit will be written to the header for
+BKGD_VAR_FLAT and NPIXELS. In the case of MIRI MRS data there are three additional
+columns in the output table:  RF_FLUX, RF_SURF_BRIGHT, and RF_BACKGROUND.
+For more details on the MIRI MRS extracted data see :ref:`MIRI-MRS-1D-residual-fringe`.
+
+Some metadata for the slit will be written to the header for
 the table extension, mostly copied from the input SCI extension headers.
 
 For slit-like modes, the extraction region is
@@ -423,10 +427,15 @@ can be applied to the flux-calibrated detector data in the :ref:`residual_fringe
 is part of the :ref:`calwebb_spec2 <calwebb_spec2>` pipeline, but currently it is skipped by default. For more
 information see :ref:`residual_fringe <residual_fringe_step>`.
 
-The pipeline also can apply a 1-D residual fringe correction. This correction is only relevant for MIRI MRS data and 
-can be turned on by setting the optional parameter `ifu_rfcorr = True`  in the ``extract_1d`` step.
+The pipeline also can apply a 1-D residual fringe correction. This correction is only relevant for MIRI MRS
+single band data. The parameter controlling applying the residual fringe correction is by default set to true, 
+`ifu_rfcorr = True`,  in the ``extract_1d`` step.
 Empirically, the 1-D correction step has been found to work better than the 2-D correction step if it is
-applied to per-band spectra.
+applied to per-band spectra. If the MIRI MRS data is from multiple bands/channels the residual fringe correction
+is turned off. Three additional columns are present in MIRI MRS extracted spectra: RF_FLUX, RF_SURF_BRIGHT, and 
+RF_BACKGROUND. These three columns are the flux, surface brightness and background arrays with the residiual
+fringe correction applied. If the data is not from a single band or the residual fringe correction fails
+NaN values are reported for the arrays. 
 
 When using the `ifu_rfcorr` option in the ``extract_1d`` step  to apply a 1-D residual fringe
 correction, it is applied during the extraction of spectra from the IFU cube. The 1D residual fringe code can also
diff --git a/jwst/extract_1d/extract_1d_step.py b/jwst/extract_1d/extract_1d_step.py
index 6e26fd9e7f..1e5fabde8c 100644
--- a/jwst/extract_1d/extract_1d_step.py
+++ b/jwst/extract_1d/extract_1d_step.py
@@ -97,7 +97,7 @@ class Extract1dStep(Step):
         and maximize the quality of the 1d spectrum. Used for IFU mode only.
     ifu_rfcorr : bool
         Switch to select whether or not to apply a 1d residual fringe correction
-        for MIRI MRS IFU spectra.  Default is False.
+        for MIRI MRS IFU spectra.  Default is True.
     ifu_set_srctype : str
         For MIRI MRS IFU data override srctype and set it to either POINT or EXTENDED.
     ifu_rscale : float
@@ -170,7 +170,7 @@ class Extract1dStep(Step):
     center_xy = float_list(min=2, max=2, default=None)  # IFU extraction x/y center
     ifu_autocen = boolean(default=False) # Auto source centering for IFU point source data.
     bkg_sigma_clip = float(default=3.0)  # background sigma clipping threshold for IFU
-    ifu_rfcorr = boolean(default=False) # Apply 1d residual fringe correction
+    ifu_rfcorr = boolean(default=True) # Apply 1d residual fringe correction (MIRI MRS only)
     ifu_set_srctype = option("POINT", "EXTENDED", None, default=None) # user-supplied source type
     ifu_rscale = float(default=None, min=0.5, max=3) # Radius in terms of PSF FWHM to scale extraction radii
     ifu_covar_scale = float(default=1.0) # Scaling factor to apply to errors to account for IFU cube covariance
@@ -342,6 +342,36 @@ def _extract_soss(self, model):
 
         return result
 
+    def _check_mrs_type(self, model):
+        """
+        Check if the MIRI MRS data is for a single band.
+
+        Parameters
+        ----------
+        model : DataModel
+            Input model.
+
+        Returns
+        -------
+        band_check : bool
+            Flag is data is for a single MIRI MRS band.
+        """
+        # check channel is only 1 value and band is only 1 value
+        validch = ["1", "2", "3", "4"]
+        validband = ["short", "medium", "long"]
+        band_check = False
+        ch_check = False
+        b_check = False
+        if model.meta.instrument.channel in validch:
+            ch_check = True
+        if str(model.meta.instrument.band).lower() in validband:
+            b_check = True
+
+        if ch_check and b_check:
+            band_check = True
+
+        return band_check
+
     def _extract_ifu(self, model, exp_type, extract_ref, apcorr_ref):
         """
         Extract IFU spectra from a single datamodel.
@@ -367,6 +397,21 @@ def _extract_ifu(self, model, exp_type, extract_ref, apcorr_ref):
             source_type = self.ifu_set_srctype
             self.log.info(f"Overriding source type and setting it to {self.ifu_set_srctype}")
 
+        if exp_type == "MIR_MRS":
+            band_cube = self._check_mrs_type(model)
+            if self.ifu_rfcorr and not band_cube:
+                message = (
+                    "Turning off residual fringe correction for MIRI MRS data "
+                    "because the input is not a single IFU band"
+                )
+                self.log.info(message)
+                self.ifu_rfcorr = False
+        else:
+            self.ifu_rfcorr = False
+            self.log.info(
+                "Turning off residual fringe correction because it only works on MIRI MRS data"
+            )
+
         result = ifu_extract1d(
             model,
             extract_ref,
diff --git a/jwst/extract_1d/ifu.py b/jwst/extract_1d/ifu.py
index dd609fc9d8..3d9f692120 100644
--- a/jwst/extract_1d/ifu.py
+++ b/jwst/extract_1d/ifu.py
@@ -107,7 +107,13 @@ def ifu_extract1d(
     else:
         log.info(f"Source type = {source_type}")
 
-    output_model = datamodels.MultiSpecModel()
+    if input_model.meta.instrument.name == "MIRI":
+        output_model = datamodels.MRSMultiSpecModel()
+        spec_dtype = datamodels.MRSSpecModel().spec_table.dtype
+    else:
+        output_model = datamodels.MultiSpecModel()
+        spec_dtype = datamodels.SpecModel().spec_table.dtype
+
     output_model.update(input_model, only="PRIMARY")
 
     slitname = input_model.meta.exposure.type
@@ -179,8 +185,13 @@ def ifu_extract1d(
     del npixels_temp
     del npixels_bkg_temp
 
+    temp_flux_rf = None
+    surf_bright_rf = None
+    background_rf = None
+
     # If selected, apply 1d residual fringe correction to the extracted spectrum
-    if (input_model.meta.instrument.name == "MIRI") & (extract_params["ifu_rfcorr"] is True):
+
+    if (input_model.meta.instrument.name == "MIRI") and (extract_params["ifu_rfcorr"] is True):
         log.info("Applying 1d residual fringe correction.")
         # Determine which MRS channel the spectrum is from
         thischannel = input_model.meta.instrument.channel
@@ -198,7 +209,7 @@ def ifu_extract1d(
 
         # Apply residual fringe to the flux array
         try:
-            temp_flux = rfutils.fit_residual_fringes_1d(
+            temp_flux_rf = rfutils.fit_residual_fringes_1d(
                 temp_flux, wavelength, channel=channel, dichroic_only=False, max_amp=None
             )
         except Exception:
@@ -206,7 +217,7 @@ def ifu_extract1d(
 
         # Apply residual fringe to the surf_bright array
         try:
-            surf_bright = rfutils.fit_residual_fringes_1d(
+            surf_bright_rf = rfutils.fit_residual_fringes_1d(
                 surf_bright, wavelength, channel=channel, dichroic_only=False, max_amp=None
             )
         except Exception:
@@ -214,7 +225,7 @@ def ifu_extract1d(
 
         # Apply residual fringe to the background array
         try:
-            background = rfutils.fit_residual_fringes_1d(
+            background_rf = rfutils.fit_residual_fringes_1d(
                 background, wavelength, channel=channel, dichroic_only=False, max_amp=None
             )
         except Exception:
@@ -230,12 +241,13 @@ def ifu_extract1d(
     f_var_poisson *= pixel_solid_angle**2 * 1.0e12  # (MJy / sr)**2 --> Jy**2
     f_var_rnoise *= pixel_solid_angle**2 * 1.0e12  # (MJy / sr)**2 --> Jy**2
     f_var_flat *= pixel_solid_angle**2 * 1.0e12  # (MJy / sr)**2 --> Jy**2
+
     # surf_bright and background were computed above
     del temp_flux
+
     error = np.sqrt(f_var_poisson + f_var_rnoise + f_var_flat)
     sb_error = np.sqrt(sb_var_poisson + sb_var_rnoise + sb_var_flat)
     berror = np.sqrt(b_var_poisson + b_var_rnoise + b_var_flat)
-    spec_dtype = datamodels.SpecModel().spec_table.dtype
 
     # If we only used the Poisson variance array as a vehicle to pass through
     # non-differentiated errors, clear it again here so that only the total
@@ -261,35 +273,80 @@ def ifu_extract1d(
         b_var_rnoise *= extract_params["ifu_covar_scale"] * extract_params["ifu_covar_scale"]
         b_var_flat *= extract_params["ifu_covar_scale"] * extract_params["ifu_covar_scale"]
 
-    otab = np.array(
-        list(
-            zip(
-                wavelength,
-                flux,
-                error,
-                f_var_poisson,
-                f_var_rnoise,
-                f_var_flat,
-                surf_bright,
-                sb_error,
-                sb_var_poisson,
-                sb_var_rnoise,
-                sb_var_flat,
-                dq,
-                background,
-                berror,
-                b_var_poisson,
-                b_var_rnoise,
-                b_var_flat,
-                npixels,
-                strict=False,
-            )
-        ),
-        dtype=spec_dtype,
-    )
+    if input_model.meta.instrument.name == "MIRI":
+        if temp_flux_rf is None:
+            flux_rf = np.full_like(flux, np.nan)
+        else:
+            flux_rf = temp_flux_rf * pixel_solid_angle * 1.0e6
+        del temp_flux_rf
+
+        if background_rf is None:
+            background_rf = np.full_like(background, np.nan)
+        if surf_bright_rf is None:
+            surf_bright_rf = np.full_like(surf_bright, np.nan)
+
+        otab = np.array(
+            list(
+                zip(
+                    wavelength,
+                    flux,
+                    error,
+                    f_var_poisson,
+                    f_var_rnoise,
+                    f_var_flat,
+                    surf_bright,
+                    sb_error,
+                    sb_var_poisson,
+                    sb_var_rnoise,
+                    sb_var_flat,
+                    dq,
+                    background,
+                    berror,
+                    b_var_poisson,
+                    b_var_rnoise,
+                    b_var_flat,
+                    npixels,
+                    flux_rf,
+                    surf_bright_rf,
+                    background_rf,
+                    strict=False,
+                )
+            ),
+            dtype=spec_dtype,
+        )
+        spec = datamodels.MRSSpecModel(spec_table=otab)
+
+    else:  # NIRSPEC
+        otab = np.array(
+            list(
+                zip(
+                    wavelength,
+                    flux,
+                    error,
+                    f_var_poisson,
+                    f_var_rnoise,
+                    f_var_flat,
+                    surf_bright,
+                    sb_error,
+                    sb_var_poisson,
+                    sb_var_rnoise,
+                    sb_var_flat,
+                    dq,
+                    background,
+                    berror,
+                    b_var_poisson,
+                    b_var_rnoise,
+                    b_var_flat,
+                    npixels,
+                    strict=False,
+                )
+            ),
+            dtype=spec_dtype,
+        )
+        spec = datamodels.SpecModel(spec_table=otab)
 
-    spec = datamodels.SpecModel(spec_table=otab)
     spec.meta.wcs = spec_wcs.create_spectral_wcs(ra, dec, wavelength)
+
     spec.spec_table.columns["wavelength"].unit = "um"
     spec.spec_table.columns["flux"].unit = "Jy"
     spec.spec_table.columns["flux_error"].unit = "Jy"
@@ -306,8 +363,14 @@ def ifu_extract1d(
     spec.spec_table.columns["bkgd_var_poisson"].unit = "(MJy/sr)^2"
     spec.spec_table.columns["bkgd_var_rnoise"].unit = "(MJy/sr)^2"
     spec.spec_table.columns["bkgd_var_flat"].unit = "(MJy/sr)^2"
+    if input_model.meta.instrument.name == "MIRI":
+        spec.spec_table.columns["rf_flux"].unit = "Jy"
+        spec.spec_table.columns["rf_surf_bright"].unit = "MJy/sr"
+        spec.spec_table.columns["rf_background"].unit = "MJy/sr"
+
     spec.slit_ra = ra
     spec.slit_dec = dec
+
     if slitname is not None and slitname != "ANY":
         spec.name = slitname
 
@@ -342,8 +405,6 @@ def ifu_extract1d(
     # See output_model.spec[0].meta.wcs instead.
     output_model.meta.wcs = None
 
-    output_model.meta.wcs = None  # See output_model.spec[i].meta.wcs instead.
-
     return output_model
 
 
diff --git a/jwst/regtest/test_miri_mrs_spec3.py b/jwst/regtest/test_miri_mrs_spec3.py
index 6e27960713..b9d4323137 100644
--- a/jwst/regtest/test_miri_mrs_spec3.py
+++ b/jwst/regtest/test_miri_mrs_spec3.py
@@ -115,7 +115,7 @@ def test_spec3_ifulong(run_spec3_ifulong, fitsdiff_default_kwargs, output):
     rtdata = run_spec3_ifulong
     rtdata.output = output
 
-    rtdata.get_truth(os.path.join(TRUTH_PATH, rtdata.output))
+    rtdata.get_truth(os.path.join(TRUTH_PATH, output))
 
     diff = FITSDiff(rtdata.output, rtdata.truth, **fitsdiff_default_kwargs)
     assert diff.identical, diff.report()
@@ -142,7 +142,7 @@ def test_spec3_ifushort(run_spec3_ifushort, fitsdiff_default_kwargs, output):
     rtdata = run_spec3_ifushort
     rtdata.output = output
 
-    rtdata.get_truth(os.path.join(TRUTH_PATH, rtdata.output))
+    rtdata.get_truth(os.path.join(TRUTH_PATH, output))
 
     diff = FITSDiff(rtdata.output, rtdata.truth, **fitsdiff_default_kwargs)
     assert diff.identical, diff.report()
@@ -169,7 +169,7 @@ def test_spec3_ifushort_emsm(run_spec3_ifushort_emsm, fitsdiff_default_kwargs, o
     rtdata = run_spec3_ifushort_emsm
     rtdata.output = output
 
-    rtdata.get_truth(os.path.join(TRUTH_PATH, rtdata.output))
+    rtdata.get_truth(os.path.join(TRUTH_PATH, output))
 
     diff = FITSDiff(rtdata.output, rtdata.truth, **fitsdiff_default_kwargs)
     assert diff.identical, diff.report()
@@ -193,7 +193,7 @@ def test_spec3_ifushort_extract1d(run_spec3_ifushort_extract1d, fitsdiff_default
     rtdata = run_spec3_ifushort_extract1d
     rtdata.output = output
 
-    rtdata.get_truth(os.path.join(TRUTH_PATH, rtdata.output))
+    rtdata.get_truth(os.path.join(TRUTH_PATH, output))
 
     diff = FITSDiff(rtdata.output, rtdata.truth, **fitsdiff_default_kwargs)
     assert diff.identical, diff.report()
diff --git a/jwst/regtest/test_miri_mrs_spec3_moving_target.py b/jwst/regtest/test_miri_mrs_spec3_moving_target.py
index 1cd7c97de9..d5e92a0fe4 100644
--- a/jwst/regtest/test_miri_mrs_spec3_moving_target.py
+++ b/jwst/regtest/test_miri_mrs_spec3_moving_target.py
@@ -47,7 +47,7 @@ def test_spec3_moving_target(run_spec3_moving_target, fitsdiff_default_kwargs, o
     rtdata = run_spec3_moving_target
     rtdata.output = output
 
-    rtdata.get_truth(os.path.join(TRUTH_PATH, rtdata.output))
+    rtdata.get_truth(os.path.join(TRUTH_PATH, output))
 
     diff = FITSDiff(rtdata.output, rtdata.truth, **fitsdiff_default_kwargs)
     assert diff.identical, diff.report()
diff --git a/jwst/regtest/test_miri_spectral_leak.py b/jwst/regtest/test_miri_spectral_leak.py
index 1a6f124e54..7e5d93acbb 100644
--- a/jwst/regtest/test_miri_spectral_leak.py
+++ b/jwst/regtest/test_miri_spectral_leak.py
@@ -11,6 +11,7 @@
 def test_miri_spectral_leak(output, rtdata, fitsdiff_default_kwargs):
     """Run cube_build on single file using coord system = ifu_align"""
 
+    # this data does not have the residual fringe corrected columns in the x1d files
     rtdata.get_asn("miri/mrs/test_spectral_leak_asn.json")
     args = ["jwst.spectral_leak.SpectralLeakStep", rtdata.input]
     Step.from_cmdline(args)
@@ -30,3 +31,33 @@ def test_miri_spectral_leak(output, rtdata, fitsdiff_default_kwargs):
 
     diff = FITSDiff(rtdata.output, rtdata.truth, **fitsdiff_default_kwargs)
     assert diff.identical, diff.report()
+
+
+@pytest.mark.bigdata
+@pytest.mark.parametrize(
+    'output',
+    ['test_spectral_leak_rf_corrected_asn_0_spectralleakstep.fits', 'test_spectral_leak_rf_corrected_asn_1_spectralleakstep.fits']
+)
+def test_miri_spectral_leak_rf(output, rtdata, fitsdiff_default_kwargs):
+    """Run cube_build on single file using coord system = ifu_align"""
+
+    # this data has the residual fringe corrected columns in the x1d files
+    rtdata.get_asn("miri/mrs/test_spectral_leak_rf_corrected_asn.json")
+    args = ["jwst.spectral_leak.SpectralLeakStep", rtdata.input]
+    Step.from_cmdline(args)
+
+    # The spectral_leak correction is part of calwebb_spec3. The top program, calwebb_spec3.py,
+    # controls writing the file names according to the association rules.
+    # For this test we just want to isolate the spectral_leak correction and run the extracted spectrum
+    # through the spectral_leak correction. This step does not have the smarts to know how to write
+    # the data. The filenames are written as:
+    # 1. test_spectral_leak_rf_corrected_asn_0_spectralleakstep.fits which is the CH1 short medium extracted spectrum (this should
+    # not be changed from the input file).
+    # 2. test_spectral_leak_rf_corrected_asn_1_spectralleakstep.fits which is the corrected CH3 short extracted spectrum
+
+    rtdata.output = f"{output}"
+    # Get the truth files
+    rtdata.get_truth(f"truth/test_miri_spectral_leak/{output}")
+
+    diff = FITSDiff(rtdata.output, rtdata.truth, **fitsdiff_default_kwargs)
+    assert diff.identical, diff.report()
diff --git a/jwst/spectral_leak/spectral_leak.py b/jwst/spectral_leak/spectral_leak.py
index 5d19499ce3..9729978b77 100644
--- a/jwst/spectral_leak/spectral_leak.py
+++ b/jwst/spectral_leak/spectral_leak.py
@@ -48,4 +48,14 @@ def do_correction(sp_leak_ref, ch1b, ch3a):
 
     # Correct the data
     spec3a_corr = spec3a - leak
-    return spec3a_corr
+
+    # now check if there exists residual fringe corrected data
+    spec3a_corr_rf = None
+
+    if isinstance(ch1b, datamodels.MRSMultiSpecModel) and isinstance(ch3a, datamodels.MRSMultiSpecModel):
+        spec1b_rf = ch1b.spec[0].spec_table.RF_FLUX
+        spec3a_rf = ch3a.spec[0].spec_table.RF_FLUX
+        leak_rf = np.interp(wave3a, 2*wave1b, spec1b_rf) * np.interp(wave3a, leak_wave, leak_percent)
+        # Correct the data
+        spec3a_corr_rf = spec3a_rf - leak_rf
+    return spec3a_corr, spec3a_corr_rf
diff --git a/jwst/spectral_leak/spectral_leak_step.py b/jwst/spectral_leak/spectral_leak_step.py
index c0319cc428..4c563591e9 100644
--- a/jwst/spectral_leak/spectral_leak_step.py
+++ b/jwst/spectral_leak/spectral_leak_step.py
@@ -49,7 +49,11 @@ def process(self, input):
 
                 for i, x1d in enumerate(input_model):  # input_model is a Model Container
                     # check that we have the correct type of data
-                    if not isinstance(x1d, datamodels.MultiSpecModel):
+                    if  isinstance(x1d, datamodels.MultiSpecModel):
+                        self.log.debug(' Data is MIRI MRS MultiSpecModel data')
+                    elif isinstance(x1d, datamodels.MRSMultiSpecModel):
+                        self.log.debug(' Data is  MIRI MRS MRSMultiSpecModel data')
+                    else:
                         self.log.warning("Data sent to spectral_leak step is not an extracted spectrum. "
                                          " It is  {}." .format(type(x1d)))
                         for r in result:
@@ -91,8 +95,10 @@ def process(self, input):
                 # done looping over data now if 1B and 3A data exists make a correction
                 # update result and return
                 if ch1b is not None and ch3a is not None:
-                    corrected_3a = spectral_leak.do_correction(sp_leak_ref, ch1b, ch3a)
+                    corrected_3a, corrected_3a_rf = spectral_leak.do_correction(sp_leak_ref, ch1b, ch3a)
                     result[ich3a].spec[0].spec_table.FLUX = corrected_3a
+                    if corrected_3a_rf is not None:
+                        result[ich3a].spec[0].spec_table.RF_FLUX = corrected_3a_rf
                     result[ich3a].meta.cal_step.spectral_leak = 'COMPLETE'
                     return result
                 else:


Reviews:
user: login: drlaw1558
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-01-14T16:30:24Z
user: login: kmacdonald-stsci
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-02-19T21:12:55Z
user: login: jemorrison
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-20T22:12:51Z
user: login: drlaw1558
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-20T22:19:25Z
user: login: melanieclarke
body: I think we're almost there!  A few minor suggestions below.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-25T21:34:42Z
user: login: jemorrison
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-25T23:21:36Z
user: login: melanieclarke
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-26T14:28:00Z
user: login: jemorrison
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-26T15:37:52Z
user: login: melanieclarke
body: Looking good!  Thanks for the new spectral_leak regtest with and without the extra columns - it's good to have that covered. 

I just have a couple more small clean up suggestions.

Regression test results look as expected for the new columns and the regtest bug fixes.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-07T14:17:21Z
user: login: jemorrison
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-07T15:22:32Z
user: login: melanieclarke
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-07T15:25:10Z
user: login: melanieclarke
body: Thanks for all the updates!  LGTM.
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-07T16:17:01Z

Review comments:
diff_hunk: @@ -172,7 +171,7 @@ class Extract1dStep(Step):
     center_xy = float_list(min=2, max=2, default=None)  # IFU extraction x/y center
     ifu_autocen = boolean(default=False) # Auto source centering for IFU point source data.
     bkg_sigma_clip = float(default=3.0)  # background sigma clipping threshold for IFU
-    ifu_rfcorr = boolean(default=False) # Apply 1d residual fringe correction
+    ifu_rfcorr = boolean(default=True) # Apply 1d residual fringe correction
path: jwst/extract_1d/extract_1d_step.py
user: login: drlaw1558
body: One concern with changing this to True is that it would then require failsafe cases for all non-MRS uses of extract1d, which may have unintended consequences.  May be easier to keep as False, since the only purpose of the flag now is to print a message saying how the flag isn't necessary any longer.
created_at: 2025-01-14T16:30:24Z
author_association: COLLABORATOR
diff_hunk: @@ -172,7 +171,7 @@ class Extract1dStep(Step):
     center_xy = float_list(min=2, max=2, default=None)  # IFU extraction x/y center
     ifu_autocen = boolean(default=False) # Auto source centering for IFU point source data.
     bkg_sigma_clip = float(default=3.0)  # background sigma clipping threshold for IFU
-    ifu_rfcorr = boolean(default=False) # Apply 1d residual fringe correction
+    ifu_rfcorr = boolean(default=True) # Apply 1d residual fringe correction
path: jwst/extract_1d/extract_1d_step.py
user: login: jemorrison
body: I have not added any print statement that says the flag is not necessary. Do we want that still ?. Since we are keeping the flag it seemed like something that might confuse the user. It never hurts to set it to True and we are not going to remove the ifu_rfcorr now. But I can add a flag that says 'By default ifu_rfcorr is true now on need to set it.' If you think it will help


Since the rf_corr is only done in ifu.py (ifu_extract1d) it only operates on MIRI and NIRSPEC. I turned off the ifu_rfcorr for all data but MIR_MRS in extract_1d_step (so NIRSPEC data).
That should do it. I ran it on a few NIRSpec IFU files I had and it works as expected.
 But I will run the regression tests to make sure

created_at: 2025-02-20T22:12:51Z
author_association: COLLABORATOR
diff_hunk: @@ -172,7 +171,7 @@ class Extract1dStep(Step):
     center_xy = float_list(min=2, max=2, default=None)  # IFU extraction x/y center
     ifu_autocen = boolean(default=False) # Auto source centering for IFU point source data.
     bkg_sigma_clip = float(default=3.0)  # background sigma clipping threshold for IFU
-    ifu_rfcorr = boolean(default=False) # Apply 1d residual fringe correction
+    ifu_rfcorr = boolean(default=True) # Apply 1d residual fringe correction
path: jwst/extract_1d/extract_1d_step.py
user: login: drlaw1558
body: Given the approach we're now taking I think this change is ok without print statements, but let's add 'for MIRI MRS data' to the comment to make clear it only affects that mode.
created_at: 2025-02-20T22:19:24Z
author_association: COLLABORATOR
diff_hunk: @@ -164,6 +170,10 @@ def ifu_extract1d(input_model, ref_file, source_type, subtract_background,
     del npixels_temp
     del npixels_bkg_temp
 
+    temp_flux_rf = None
+    surf_bright_rf = None
+    background_rf = None
+
     # If selected, apply 1d residual fringe correction to the extracted spectrum
     if ((input_model.meta.instrument.name == 'MIRI') & (extract_params['ifu_rfcorr'] is True)):
path: jwst/extract_1d/ifu.py
user: login: melanieclarke
body: This should be the logical `and`, not the bitwise `&`

```suggestion
    if ((input_model.meta.instrument.name == 'MIRI') and (extract_params['ifu_rfcorr'] is True)):
```
created_at: 2025-02-25T21:15:04Z
author_association: COLLABORATOR
diff_hunk: @@ -208,17 +219,32 @@ def ifu_extract1d(input_model, ref_file, source_type, subtract_background,
                     "the flux will not be correct.")
         pixel_solid_angle = 1.
 
-    # Convert flux from MJy / steradian to Jy.
+    # Convert flux from MJy / steradian to Jy
     flux = temp_flux * pixel_solid_angle * 1.e6
     f_var_poisson *= (pixel_solid_angle ** 2 * 1.e12)  # (MJy / sr)**2 --> Jy**2
     f_var_rnoise *= (pixel_solid_angle ** 2 * 1.e12)  # (MJy / sr)**2 --> Jy**2
     f_var_flat *= (pixel_solid_angle ** 2 * 1.e12)  # (MJy / sr)**2 --> Jy**2
     # surf_bright and background were computed above
+
+    if temp_flux_rf is None:
+        flux_rf = np.zeros_like(flux)
path: jwst/extract_1d/ifu.py
user: login: melanieclarke
body: These arrays are only needed for MIRI, so they should probably be moved down into the 'if MIRI' block.
created_at: 2025-02-25T21:16:32Z
author_association: COLLABORATOR
diff_hunk: @@ -107,6 +107,12 @@ def ifu_extract1d(input_model, ref_file, source_type, subtract_background,
         log.info(f"Source type = {source_type}")
 
     output_model = datamodels.MultiSpecModel()
+    spec_dtype = datamodels.SpecModel().spec_table.dtype
+
+    if input_model.meta.instrument.name == 'MIRI':
+        output_model = datamodels.MRSMultiSpecModel()
+        spec_dtype = datamodels.MRSSpecModel().spec_table.dtype
path: jwst/extract_1d/ifu.py
user: login: melanieclarke
body: This should probably be an if/else, rather than creating the output_model and then immediately overwriting it for MIRI.
created_at: 2025-02-25T21:17:35Z
author_association: COLLABORATOR
diff_hunk: @@ -340,13 +339,41 @@ def _extract_soss(self, model):
 
         return result
 
+    def _check_mrs_type(self,model):
+        """ Check if the MIRI MRS data is for a single band """
+        # check channel is only 1 value and band is only 1 value
+        validch = ['1', '2', '3', '4']
+        validband = ['short', 'medium', 'long']
+        band_check = False
+        ch_check = False
+        b_check = False
+        if model.meta.instrument.channel  in validch:
+            ch_check = True
+        if model.meta.instrument.band.lower()  in validband:
+            b_check = True
+
+        if ch_check and b_check:
+            band_check = True
+
+        return band_check
+
     def _extract_ifu(self, model, exp_type, extract_ref, apcorr_ref):
         """Extract IFU spectra from a single datamodel."""
         source_type = model.meta.target.source_type
         if self.ifu_set_srctype is not None and exp_type == 'MIR_MRS':
             source_type = self.ifu_set_srctype
             self.log.info(f"Overriding source type and setting it to {self.ifu_set_srctype}")
 
+        if exp_type == 'MIR_MRS':
+            band_cube = self._check_mrs_type(model)
+            if self.ifu_rfcorr and  not band_cube:
path: jwst/extract_1d/extract_1d_step.py
user: login: melanieclarke
body: ```suggestion
            if self.ifu_rfcorr and not band_cube:
```
created_at: 2025-02-25T21:19:57Z
author_association: COLLABORATOR
diff_hunk: @@ -340,13 +339,41 @@ def _extract_soss(self, model):
 
         return result
 
+    def _check_mrs_type(self,model):
+        """ Check if the MIRI MRS data is for a single band """
+        # check channel is only 1 value and band is only 1 value
+        validch = ['1', '2', '3', '4']
+        validband = ['short', 'medium', 'long']
+        band_check = False
+        ch_check = False
+        b_check = False
+        if model.meta.instrument.channel  in validch:
+            ch_check = True
+        if model.meta.instrument.band.lower()  in validband:
path: jwst/extract_1d/extract_1d_step.py
user: login: melanieclarke
body: Add a safety check here for `None` in the band value:

```suggestion
        if str(model.meta.instrument.band).lower() in validband:
```
I think this is causing the current unit test failures.
created_at: 2025-02-25T21:21:33Z
author_association: COLLABORATOR
diff_hunk: @@ -77,7 +77,11 @@ be an output table extension with the name EXTRACT1D.  This extension will
 have columns WAVELENGTH, FLUX, FLUX_ERROR, FLUX_VAR_POISSON, FLUX_VAR_RNOISE,
 FLUX_VAR_FLAT, SURF_BRIGHT, SB_ERROR, SB_VAR_POISSON, SB_VAR_RNOISE,
 SB_VAR_FLAT, DQ, BACKGROUND, BKGD_ERROR, BKGD_VAR_POISSON, BKGD_VAR_RNOISE,
-BKGD_VAR_FLAT and NPIXELS. Some metadata for the slit will be written to the header for
+BKGD_VAR_FLAT and NPIXELS. In the case of MIRI MRS data there are three additional
+columns in the output table:  RF_FLUX, RF_SURF_BRIGHT, and  RF_BACKGROUND
path: docs/jwst/extract_1d/description.rst
user: login: melanieclarke
body: Remove extra space, add period

```suggestion
columns in the output table:  RF_FLUX, RF_SURF_BRIGHT, and RF_BACKGROUND.
```
created_at: 2025-02-25T21:22:14Z
author_association: COLLABORATOR
diff_hunk: @@ -423,10 +427,15 @@ can be applied to the flux-calibrated detector data in the :ref:`residual_fringe
 is part of the :ref:`calwebb_spec2 <calwebb_spec2>` pipeline, but currently it is skipped by default. For more
 information see :ref:`residual_fringe <residual_fringe_step>`.
 
-The pipeline also can apply a 1-D residual fringe correction. This correction is only relevant for MIRI MRS data and 
-can be turned on by setting the optional parameter `ifu_rfcorr = True`  in the ``extract_1d`` step.
+The pipeline also can apply a 1-D residual fringe correction. This correction is only relevant for MIRI MRS
+single band data. The parameter controlling applying the residual fringe correction is by default set to true, 
+`ifu_rfcorr = True`,  in the ``extract_1d`` step.
 Empirically, the 1-D correction step has been found to work better than the 2-D correction step if it is
-applied to per-band spectra.
+applied to per-band spectra. If the MIRI MRS data is from multiple bands/channels the residual fringe correction
+is turned off. Three additional columns are present in MIRI MRS extracted spectra: RF_FLUX, RF_SURF_BRIGHT, and 
+RF_BACKGROUND  These three columns are the flux, surface brightness and background arrays with the residiual
path: docs/jwst/extract_1d/description.rst
user: login: melanieclarke
body: Add a period:
```suggestion
RF_BACKGROUND. These three columns are the flux, surface brightness and background arrays with the residiual
```
created_at: 2025-02-25T21:32:37Z
author_association: COLLABORATOR
diff_hunk: @@ -423,10 +427,15 @@ can be applied to the flux-calibrated detector data in the :ref:`residual_fringe
 is part of the :ref:`calwebb_spec2 <calwebb_spec2>` pipeline, but currently it is skipped by default. For more
 information see :ref:`residual_fringe <residual_fringe_step>`.
 
-The pipeline also can apply a 1-D residual fringe correction. This correction is only relevant for MIRI MRS data and 
-can be turned on by setting the optional parameter `ifu_rfcorr = True`  in the ``extract_1d`` step.
+The pipeline also can apply a 1-D residual fringe correction. This correction is only relevant for MIRI MRS
+single band data. The parameter controlling applying the residual fringe correction is by default set to true, 
+`ifu_rfcorr = True`,  in the ``extract_1d`` step.
 Empirically, the 1-D correction step has been found to work better than the 2-D correction step if it is
-applied to per-band spectra.
+applied to per-band spectra. If the MIRI MRS data is from multiple bands/channels the residual fringe correction
+is turned off. Three additional columns are present in MIRI MRS extracted spectra: RF_FLUX, RF_SURF_BRIGHT, and 
+RF_BACKGROUND  These three columns are the flux, surface brightness and background arrays with the residiual
+fringe correction applied. If the data is not from a single band or the residual fringe correction fails
+NULL values are reported for the arrays. 
path: docs/jwst/extract_1d/description.rst
user: login: melanieclarke
body: We should be explicit here - the arrays are filled with NaN values.
created_at: 2025-02-25T21:33:22Z
author_association: COLLABORATOR
diff_hunk: @@ -423,10 +427,15 @@ can be applied to the flux-calibrated detector data in the :ref:`residual_fringe
 is part of the :ref:`calwebb_spec2 <calwebb_spec2>` pipeline, but currently it is skipped by default. For more
 information see :ref:`residual_fringe <residual_fringe_step>`.
 
-The pipeline also can apply a 1-D residual fringe correction. This correction is only relevant for MIRI MRS data and 
-can be turned on by setting the optional parameter `ifu_rfcorr = True`  in the ``extract_1d`` step.
+The pipeline also can apply a 1-D residual fringe correction. This correction is only relevant for MIRI MRS
+single band data. The parameter controlling applying the residual fringe correction is by default set to true, 
+`ifu_rfcorr = True`,  in the ``extract_1d`` step.
 Empirically, the 1-D correction step has been found to work better than the 2-D correction step if it is
-applied to per-band spectra.
+applied to per-band spectra. If the MIRI MRS data is from multiple bands/channels the residual fringe correction
+is turned off. Three additional columns are present in MIRI MRS extracted spectra: RF_FLUX, RF_SURF_BRIGHT, and 
+RF_BACKGROUND  These three columns are the flux, surface brightness and background arrays with the residiual
+fringe correction applied. If the data is not from a single band or the residual fringe correction fails
+NULL values are reported for the arrays. 
path: docs/jwst/extract_1d/description.rst
user: login: jemorrison
body: I have looked at the tables and they do not have NaN but NULL.  
They are set to NaN but when they get written to the FITS table they are turned into NULL.
Any problem with that ?

created_at: 2025-02-25T23:21:36Z
author_association: COLLABORATOR
diff_hunk: @@ -423,10 +427,15 @@ can be applied to the flux-calibrated detector data in the :ref:`residual_fringe
 is part of the :ref:`calwebb_spec2 <calwebb_spec2>` pipeline, but currently it is skipped by default. For more
 information see :ref:`residual_fringe <residual_fringe_step>`.
 
-The pipeline also can apply a 1-D residual fringe correction. This correction is only relevant for MIRI MRS data and 
-can be turned on by setting the optional parameter `ifu_rfcorr = True`  in the ``extract_1d`` step.
+The pipeline also can apply a 1-D residual fringe correction. This correction is only relevant for MIRI MRS
+single band data. The parameter controlling applying the residual fringe correction is by default set to true, 
+`ifu_rfcorr = True`,  in the ``extract_1d`` step.
 Empirically, the 1-D correction step has been found to work better than the 2-D correction step if it is
-applied to per-band spectra.
+applied to per-band spectra. If the MIRI MRS data is from multiple bands/channels the residual fringe correction
+is turned off. Three additional columns are present in MIRI MRS extracted spectra: RF_FLUX, RF_SURF_BRIGHT, and 
+RF_BACKGROUND  These three columns are the flux, surface brightness and background arrays with the residiual
+fringe correction applied. If the data is not from a single band or the residual fringe correction fails
+NULL values are reported for the arrays. 
path: docs/jwst/extract_1d/description.rst
user: login: melanieclarke
body: You mean the string "NULL"? How were you looking at them? Reading in an example with either astropy.io.fits or with datamodels, I see float64 NaN values, which should be correct.
created_at: 2025-02-26T14:27:59Z
author_association: COLLABORATOR
diff_hunk: @@ -423,10 +427,15 @@ can be applied to the flux-calibrated detector data in the :ref:`residual_fringe
 is part of the :ref:`calwebb_spec2 <calwebb_spec2>` pipeline, but currently it is skipped by default. For more
 information see :ref:`residual_fringe <residual_fringe_step>`.
 
-The pipeline also can apply a 1-D residual fringe correction. This correction is only relevant for MIRI MRS data and 
-can be turned on by setting the optional parameter `ifu_rfcorr = True`  in the ``extract_1d`` step.
+The pipeline also can apply a 1-D residual fringe correction. This correction is only relevant for MIRI MRS
+single band data. The parameter controlling applying the residual fringe correction is by default set to true, 
+`ifu_rfcorr = True`,  in the ``extract_1d`` step.
 Empirically, the 1-D correction step has been found to work better than the 2-D correction step if it is
-applied to per-band spectra.
+applied to per-band spectra. If the MIRI MRS data is from multiple bands/channels the residual fringe correction
+is turned off. Three additional columns are present in MIRI MRS extracted spectra: RF_FLUX, RF_SURF_BRIGHT, and 
+RF_BACKGROUND  These three columns are the flux, surface brightness and background arrays with the residiual
+fringe correction applied. If the data is not from a single band or the residual fringe correction fails
+NULL values are reported for the arrays. 
path: docs/jwst/extract_1d/description.rst
user: login: jemorrison
body: A quick way to look at the extract1d Fits tables is to use the FV package by nasa. It displays the table in an easily readable form. But if the data is NaN reading with astroy py and using datamodels then I will just change the documentation to to reflect that
created_at: 2025-02-26T15:37:52Z
author_association: COLLABORATOR
diff_hunk: @@ -170,7 +170,7 @@ class Extract1dStep(Step):
     center_xy = float_list(min=2, max=2, default=None)  # IFU extraction x/y center
     ifu_autocen = boolean(default=False) # Auto source centering for IFU point source data.
     bkg_sigma_clip = float(default=3.0)  # background sigma clipping threshold for IFU
-    ifu_rfcorr = boolean(default=False) # Apply 1d residual fringe correction
+    ifu_rfcorr = boolean(default=True) # Apply 1d residual fringe correction
path: jwst/extract_1d/extract_1d_step.py
user: login: melanieclarke
body: David's comment here still needs addressing - please add a comment about MIRI MRS here.  E.g.:
```suggestion
    ifu_rfcorr = boolean(default=True) # Apply 1d residual fringe correction (MIRI MRS only)```
created_at: 2025-03-07T13:29:07Z
author_association: COLLABORATOR
diff_hunk: @@ -261,35 +273,84 @@ def ifu_extract1d(
         b_var_rnoise *= extract_params["ifu_covar_scale"] * extract_params["ifu_covar_scale"]
         b_var_flat *= extract_params["ifu_covar_scale"] * extract_params["ifu_covar_scale"]
 
-    otab = np.array(
-        list(
-            zip(
-                wavelength,
-                flux,
-                error,
-                f_var_poisson,
-                f_var_rnoise,
-                f_var_flat,
-                surf_bright,
-                sb_error,
-                sb_var_poisson,
-                sb_var_rnoise,
-                sb_var_flat,
-                dq,
-                background,
-                berror,
-                b_var_poisson,
-                b_var_rnoise,
-                b_var_flat,
-                npixels,
-                strict=False,
-            )
-        ),
-        dtype=spec_dtype,
-    )
+    if input_model.meta.instrument.name == "MIRI":
+        if temp_flux_rf is None:
+            flux_rf = np.zeros_like(flux)
+            flux_rf[:] = np.nan
+        else:
+            flux_rf = temp_flux_rf * pixel_solid_angle * 1.0e6
+        del temp_flux_rf
+
+        if background_rf is None:
+            background_rf = np.zeros_like(background)
+            background_rf[:] = np.nan
+
+        if surf_bright_rf is None:
+            surf_bright_rf = np.zeros_like(surf_bright)
+            surf_bright_rf[:] = np.nan
path: jwst/extract_1d/ifu.py
user: login: melanieclarke
body: These array creation commands would be simpler with np.full_like:
```suggestion
        if temp_flux_rf is None:
            flux_rf = np.full_like(flux, np.nan)
        else:
            flux_rf = temp_flux_rf * pixel_solid_angle * 1.0e6
        del temp_flux_rf

        if background_rf is None:
            background_rf = np.full_like(background, np.nan)

        if surf_bright_rf is None:
            surf_bright_rf = np.full_like(surf_bright, np.nan)
```
created_at: 2025-03-07T13:35:15Z
author_association: COLLABORATOR
diff_hunk: @@ -342,6 +342,36 @@ def _extract_soss(self, model):
 
         return result
 
+    def _check_mrs_type(self, model):
+        """
+        Check if the MIRI MRS data is for a single band.
+
+        Parameters
+        ----------
+        model : DataModel
+            Input model.
+
+        Returns
+        -------
+        band_check : logical
path: jwst/extract_1d/extract_1d_step.py
user: login: melanieclarke
body: ```suggestion
        band_check : bool
```
created_at: 2025-03-07T13:42:29Z
author_association: COLLABORATOR
diff_hunk: @@ -74,7 +74,7 @@ def ifu_extract1d(
         in IFU mode.  Default is False.
     ifu_rfcorr : bool, optional
         Switch to select whether or not to apply a 1d residual fringe correction
-        for MIRI MRS IFU spectra.  Default is False.
+        for MIRI MRS IFU spectra.  Default is True.
path: jwst/extract_1d/ifu.py
user: login: melanieclarke
body: It looks like the default value for this function is still set to False - we should update it.
created_at: 2025-03-07T13:44:09Z
author_association: COLLABORATOR
diff_hunk: @@ -74,7 +74,7 @@ def ifu_extract1d(
         in IFU mode.  Default is False.
     ifu_rfcorr : bool, optional
         Switch to select whether or not to apply a 1d residual fringe correction
-        for MIRI MRS IFU spectra.  Default is False.
+        for MIRI MRS IFU spectra.  Default is True.
path: jwst/extract_1d/ifu.py
user: login: jemorrison
body: See if this works for  logic on ifu_rfcorr:

In extract_1d_step ifu_rfcorr is set to True
It can get set to false if the input data is not single band data. 
Which should cover the general pipeline logic

But if we set it True in ifu.py I think that would still work. It will get turned off if it not single band data extract_1d_step. 

created_at: 2025-03-07T15:22:32Z
author_association: COLLABORATOR
diff_hunk: @@ -74,7 +74,7 @@ def ifu_extract1d(
         in IFU mode.  Default is False.
     ifu_rfcorr : bool, optional
         Switch to select whether or not to apply a 1d residual fringe correction
-        for MIRI MRS IFU spectra.  Default is False.
+        for MIRI MRS IFU spectra.  Default is True.
path: jwst/extract_1d/ifu.py
user: login: melanieclarke
body: It should get overridden with the appropriate value from the step in any real use case.  But you're right that it's probably safer to default to False here, since this function doesn't have the mode-checking logic. 

In that case - let's just change that comment back to "Default is False."
created_at: 2025-03-07T15:25:10Z
author_association: COLLABORATOR

PR comments:
user: login: codecov[bot]
created_at: 2025-01-13T22:40:47Z
body: ## [Codecov](https://app.codecov.io/gh/spacetelescope/jwst/pull/9073?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) Report
Attention: Patch coverage is `57.81250%` with `27 lines` in your changes missing coverage. Please review.
> Project coverage is 72.51%. Comparing base [(`c8cf2ec`)](https://app.codecov.io/gh/spacetelescope/jwst/commit/c8cf2ec2785e94723088b3723d576760a5a887ad?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) to head [(`21f55f9`)](https://app.codecov.io/gh/spacetelescope/jwst/commit/21f55f98c7206eb4dfaec2e446eebf7d718d36ed?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope).
> Report is 793 commits behind head on main.

| [Files with missing lines](https://app.codecov.io/gh/spacetelescope/jwst/pull/9073?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) | Patch % | Lines |
|---|---|---|
| [jwst/extract\_1d/ifu.py](https://app.codecov.io/gh/spacetelescope/jwst/pull/9073?src=pr&el=tree&filepath=jwst%2Fextract_1d%2Fifu.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope#diff-andzdC9leHRyYWN0XzFkL2lmdS5weQ==) | 72.41% | [8 Missing :warning: ](https://app.codecov.io/gh/spacetelescope/jwst/pull/9073?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) |
| [jwst/spectral\_leak/spectral\_leak.py](https://app.codecov.io/gh/spacetelescope/jwst/pull/9073?src=pr&el=tree&filepath=jwst%2Fspectral_leak%2Fspectral_leak.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope#diff-andzdC9zcGVjdHJhbF9sZWFrL3NwZWN0cmFsX2xlYWsucHk=) | 0.00% | [7 Missing :warning: ](https://app.codecov.io/gh/spacetelescope/jwst/pull/9073?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) |
| [jwst/spectral\_leak/spectral\_leak\_step.py](https://app.codecov.io/gh/spacetelescope/jwst/pull/9073?src=pr&el=tree&filepath=jwst%2Fspectral_leak%2Fspectral_leak_step.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope#diff-andzdC9zcGVjdHJhbF9sZWFrL3NwZWN0cmFsX2xlYWtfc3RlcC5weQ==) | 0.00% | [7 Missing :warning: ](https://app.codecov.io/gh/spacetelescope/jwst/pull/9073?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) |
| [jwst/extract\_1d/extract\_1d\_step.py](https://app.codecov.io/gh/spacetelescope/jwst/pull/9073?src=pr&el=tree&filepath=jwst%2Fextract_1d%2Fextract_1d_step.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope#diff-andzdC9leHRyYWN0XzFkL2V4dHJhY3RfMWRfc3RlcC5weQ==) | 76.19% | [5 Missing :warning: ](https://app.codecov.io/gh/spacetelescope/jwst/pull/9073?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #9073      +/-   ##
==========================================
- Coverage   73.71%   72.51%   -1.21%     
==========================================
  Files         369      371       +2     
  Lines       36457    37092     +635     
==========================================
+ Hits        26875    26896      +21     
- Misses       9582    10196     +614     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/spacetelescope/jwst/pull/9073?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: jemorrison
created_at: 2025-02-19T16:03:32Z
body: @drlaw1558 I think this PR is was is needed. I am going to run the regression tests to make sure all the NIRSPEC data is the same. 
author_association: COLLABORATOR
user: login: drlaw1558
created_at: 2025-02-19T19:20:50Z
body: @jemorrison Looks like a couple issues at the moment.
- rf_flux (and other rf quantities) aren't getting calculated unless the step is passed ifu_rfcorr=True.  We want this to run all the time for MRS spectra, ignoring the ifu_rfcorr setting.
- All that the ifu_rfcorr flag should do is to print a message saying that the flag does nothing any longer, and that RF-corrected spectra are automatically present in the x1d data.
In a later build we'd then remove the ifu_rfcorr step option, we're just going through a temporary grace period to allow folks to adapt to the new usage so that their notebooks don't crash if they try to set a no-longer existing option.
author_association: COLLABORATOR
user: login: jemorrison
created_at: 2025-02-19T20:28:37Z
body: @drlaw1558 I am assuming you do not want the rf correction run on NON-single band data. So in spec2 or if a user created multi band cubes. That is why I left  to turn on and off. I was thinking the parameter reference file in spec3 would turn it on.


That seems the cleanest. Are you thinking something else?

author_association: COLLABORATOR
user: login: drlaw1558
created_at: 2025-02-19T23:01:21Z
body: @jemorrison Gotcha; my original aim was ultimately to be able to retire the flag entirely, so that users never need to see or interact with it.  The rfcorr columns would just be something that may or may not be better than the regular columns (depending on how well the algorithm performs in each case).  The existing code already can cleanly handle the case of being passed per-channel or all-wavelength cubes, returning unaltered vectors, and we could make sure that it exits cleanly in the spec2 cube case as well though.

I see the point that you're making though; we could keep the ifu_rfcorr argument and just set it to True by default in spec3 param files.  One problem with that though is that this wouldn't cover the use case of rerunning just Extract1D on cubes that spec3 already created (which I certainly use in a number of notebooks).  We could add ifu_rfcorr=True to the extract1d param file, but that would then try to turn it on for spec2 version of extract1d too (or users running Extract1D on spec2 cubes).  That might be ok though since there's a block in the code to set it to FALSE for spec2 cubes...

Thoughts?
author_association: COLLABORATOR
user: login: jemorrison
created_at: 2025-02-20T00:05:01Z
body: @drlaw1558 We could  just do as you first suggested and run the residual fringe correction on all data and populate the 3 columns of the x1d. We are going to have to update the documentation to describe these new columns. We could put the information in there that residual corrected columns are really only for single band cubes. I do worry people will miss that information.

Or we could set rf_corr to true in extract1d step (default = True)  or the parameter file as you suggest and allow the code to determine when it should not be run (i.e. the band is not single).  With a clean log message that say turning off residual fringe correction and add that information to the docs. 

I say pick the one you think will be the clearest/easiest  for the users
author_association: COLLABORATOR
user: login: melanieclarke
created_at: 2025-02-20T15:07:00Z
body: > Or we could set rf_corr to true in extract1d step (default = True) or the parameter file as you suggest and allow the code to determine when it should not be run (i.e. the band is not single). With a clean log message that say turning off residual fringe correction and add that information to the docs.

This would be my preference if possible: ignore the flag as David first suggested and determine inside the code if it's possible to usefully run the rfcorr algorithm.  If so, run it and store the results in the new columns.  If not, store NaNs.
author_association: COLLABORATOR
user: login: jemorrison
created_at: 2025-02-20T17:45:04Z
body: @melanieclarke @drlaw1558 
I think this works:
rf_corr=True by default in extract_1d_step.py. 
The extract_1d_step will check if the MRS data is a single band. If it is not then rf_corr is set to false
and  prints a log message saying it is turned off is printed. 
If ifu_rfcorr is false then the 3 new columns in the spec table are set to NULL.

This will allow the user to run extract_1d on an IFUcube and the program determines if the residual fringe correction can be run and they do not need worry about setting it to false. If they did set it to false then the columns in the spec table will be NULL.

Does this work for all the cases ?

author_association: COLLABORATOR
user: login: drlaw1558
created_at: 2025-02-20T18:27:46Z
body: @jemorrison I think that works for me; no need to update param ref files and we keep the existing logic flow.  Anyone working with MRS data who doesn't override it will get rf-corrected spectra automatically in the cases where it's possible to create them.

We should ensure though that we've caught all of the cases where rfcorr needs to be disabled within the code.  I.e., check that it's turned off for everything except MRS data (if we aren't already).  We should also be sure that we catch the failure cases.

I can see one potential problem in the code; the flux, surf_bright, and background calls to residual fringe are all embedded in their own try/except blocks, but we only have a catch below for if the temp_flux_rf calculation failed.  If that calculation succeeds but the surf_bright or background calculations fail, then the zip statement down below will fail.  In the ton of data that will now be going through this routine, I'm sure that condition will occur from time to time.  So, we should have some kind of catch for whenever any of those three fail out.
author_association: COLLABORATOR
user: login: jemorrison
created_at: 2025-02-25T17:11:34Z
body: @drlaw1558 Good catch on that possible error of the way the try except was coded. 
Now all 3 arrays are checked if one of them is None - then all are set to Nan. 
author_association: COLLABORATOR
user: login: jemorrison
created_at: 2025-02-25T17:21:40Z
body: @melanieclarke ready for your review. 
Starting regression tests now too

author_association: COLLABORATOR
user: login: drlaw1558
created_at: 2025-02-25T17:24:21Z
body: @jemorrison How about just setting the ones that fail to NaN?  That way we might still get a useful de-fringed source spectrum even if the background spectrum defringing runs into problems.
author_association: COLLABORATOR
user: login: jemorrison
created_at: 2025-02-25T17:33:33Z
body: @drlaw1558 Ok I will do that. 
author_association: COLLABORATOR
user: login: melanieclarke
created_at: 2025-02-26T15:05:20Z
body: @jemorrison - it just occurred to me that since MIRI MRS extracted spectra now have a different datamodel, there is more follow up work outside extract1d that needs to be done.  We need to make sure any step that takes in a MultiSpecModel can also read a MRSMultiSpecModel if needed.  

For example, running the spec3 regression tests, it looks like `spectral_leak` is currently being skipped with this warning:
`WARNING - Data sent to spectral_leak step is not an extracted spectrum.  It is  <class 'stdatamodels.jwst.datamodels.multispec.MRSMultiSpecModel'>`

Also, the `combine_1d` step succeeds, but does not handle the additional columns.
author_association: COLLABORATOR
user: login: melanieclarke
created_at: 2025-02-26T16:13:46Z
body: And one more thing - running regtests locally, it looks like there is a sneaky bug in the regression tests in test_miri_mrs_spec3.py.  They are reporting no differences, despite creating a totally different datamodel with 3 new columns!

I think I tracked it down to this line:
`rtdata.get_truth(os.path.join(TRUTH_PATH, rtdata.output))`

It needs to be this:
`rtdata.get_truth(os.path.join(TRUTH_PATH, output))`

The first line copies the newly created output file as the truth and therefore finds no differences.  The second one appropriately retrieves the truth file from artifactory and then finds the expected differences.
author_association: COLLABORATOR
user: login: jemorrison
created_at: 2025-02-26T17:40:15Z
body: @melanieclarke 
Just what to make sure I understand the change needed for the regtest test_miri_mrs_spec3.py
The line before rtdata.get_truth sets
rtdata.output = output 
Every test in test_miri_mrs_spec3.py works like this.
So what I am not understanding ?
author_association: COLLABORATOR
user: login: melanieclarke
created_at: 2025-02-26T17:45:19Z
body: > @melanieclarke Just what to make sure I understand the change needed for the regtest test_miri_mrs_spec3.py The line before rtdata.get_truth sets rtdata.output = output Every test in test_miri_mrs_spec3.py works like this. So what I am not understanding ?

That's the sneaky part -- when rtdata.output is set, it automatically modifies the input value to turn it into an absolute path.  Even though it looks like `rtdata.output` equals `output`, it doesn't.  All of the tests in test_miri_mrs_spec3.py and test_miri_mrs_spec3_moving_target.py need to be changed to use the relative path in `output` instead of the absolute path in `rtdata.output`.
author_association: COLLABORATOR
user: login: jemorrison
created_at: 2025-02-26T17:56:41Z
body: @melanieclarke I have a bunch of conflicts now with ifu.py and I made many commits.
How do I avoid having to change every commit and only do it for the last one ?
I think that is either how you or Ken suggested doing it
author_association: COLLABORATOR
user: login: melanieclarke
created_at: 2025-02-26T18:05:26Z
body: > How do I avoid having to change every commit and only do it for the last one ?

If it's too many or too difficult to rebase, you can just merge instead and resolve the conflicts once.
author_association: COLLABORATOR
user: login: jemorrison
created_at: 2025-02-26T18:11:09Z
body: So do you first do a 'git fetch jwst '
then 'git merge '
author_association: COLLABORATOR
user: login: jemorrison
created_at: 2025-03-03T16:53:17Z
body: @drlaw1558 I am updating the spectral_leal correction to work on residual fringe corrected data
Do we need to catch that a user might trying to run spec3 on data processed earlier so there does not exist the columns for
residual fringe corrected data  ?
author_association: COLLABORATOR
user: login: drlaw1558
created_at: 2025-03-03T21:25:57Z
body: @jemorrison Wouldn't hurt to check for it, users can do many unexpected things...
author_association: COLLABORATOR
user: login: jemorrison
created_at: 2025-03-04T22:57:26Z
body: @drlaw1558 The masterbackground step reads in the FLUX column from the spec table. If we want that step to use the residual fringe corrected flux we will need to add that. 

@melanieclarke I think both master background and combine1d will run fine because datamodels.open is used to open the x1d files. They are not assumed to be a MultiSpecModel. The documentation has assumed they are MultiSpecModel. The code will run (will confirm with regression testing) the FLUX column is used to access to residual fringe corrected data for master background. But that is probably fine - for now. 
author_association: COLLABORATOR
user: login: drlaw1558
created_at: 2025-03-04T23:03:28Z
body: @jemorrison We want the master_background step to continue reading the FLUX column since the subtraction will happen in cube space.
author_association: COLLABORATOR
user: login: jemorrison
created_at: 2025-03-05T17:28:28Z
body: regression tests:
https://github.com/spacetelescope/RegressionTests/actions/runs/13664659037
author_association: COLLABORATOR
user: login: jemorrison
created_at: 2025-03-06T22:14:50Z
body: @tapastro @melanieclarke This PR is ready for a final review. 
The regression tests located at:https://github.com/spacetelescope/RegressionTests/actions/runs/13664659037 
have the expected failures of the 3 additional columns in the x1d data for the MRS. Since the moving target regression test was updated and a bug in testing the files was removed there are moving target failures.

author_association: COLLABORATOR