Repository: spacetelescope/jwst

Issue:
Updates to PHOTOM step and files for MIRI imager time-dependent throughput
_Issue [JP-3455](https://jira.stsci.edu/browse/JP-3455) was created on JIRA by [Misty Cracraft](https://jira.stsci.edu/secure/ViewProfile.jspa?name=cracraft):_

A time-dependent throughput issue was found for the MIRI Imager, as had previously been found for the MIRI MRS. As with the fix for the MIRI MRS, we now have updated code that can go into the pipeline to use updated photom files with a correction based on MJD of the observation taken from the header and will not rely on a set of USEAFTER date specified photom files like we currently have in place for the workaround.

[Mattia Libralato](https://jira.stsci.edu/secure/ViewProfile.jspa?name=libra)  has developed the code and created the appropriate photom reference file for this.

 

Discussion:
mlibralato: @hbushouse Let me know if everything works now.
hbushouse: All of the CI failures are due to errors in the jwst/photom/tests/test_photom.py unit test module when it tries to access the new "timecoeff" attribute in the ref file model, but it's not defined here, due to the fact that the CI test environment is still using the latest stdatamodels release, rather than stdatamodels/master.

@mlibralato Can you try to verify that running "pytest test_photom.py" in your local branch, which presumably has the latest stdatamodels updates, succeeds?
hbushouse: Also, I still can't run the full regression test suite on this PR branch due to some kind of password protection or permission issue with your fork of the repo. So once you've made the suggested changes on your local PR branch, can you at least run a few of the regression test modules manually, just so we have some kind of assurance that this works OK? To do so, go to the /jwst/jwst/regtest directory in your local branch and execute the MIRI imager tests using:

`pytest --bigdata test_miri_image.py`

Note that you have to first setup the environment variable `export TEST_BIGDATA=https://bytesalad.stsci.edu/artifactory` so that Jenkins knows where to get the input data files used by the tests.
mlibralato: > All of the CI failures are due to errors in the jwst/photom/tests/test_photom.py unit test module when it tries to access the new "timecoeff" attribute in the ref file model, but it's not defined here, due to the fact that the CI test environment is still using the latest stdatamodels release, rather than stdatamodels/master.
> 
> @mlibralato Can you try to verify that running "pytest test_photom.py" in your local branch, which presumably has the latest stdatamodels updates, succeeds?

I tested the code before pushing it and with the updated stdatamodels I get a "pass".
hbushouse: > > Also, I still can't run the full regression test suite on this PR branch due to some kind of password protection or permission issue with your fork of the repo. So once you've made the suggested changes on your local PR branch, can you at least run a few of the regression test modules manually, just so we have some kind of assurance that this works OK? To do so, go to the /jwst/jwst/regtest directory in your local branch and execute the MIRI imager tests using:
> > `pytest --bigdata test_miri_image.py`
> > Note that you have to first setup the environment variable `export TEST_BIGDATA=https://bytesalad.stsci.edu/artifactory` so that Jenkins knows where to get the input data files used by the tests.
> 
> The test failed but I wonder if all errors are due to the change in the PHOTOM step itself.

That's probably the case. I noticed that the differences start with the "cal" product and proceed downstream from there, which is exactly what you'd expect from changes in the photom correction (which is the last step before "cal" is created).
mlibralato: > > > Also, I still can't run the full regression test suite on this PR branch due to some kind of password protection or permission issue with your fork of the repo. So once you've made the suggested changes on your local PR branch, can you at least run a few of the regression test modules manually, just so we have some kind of assurance that this works OK? To do so, go to the /jwst/jwst/regtest directory in your local branch and execute the MIRI imager tests using:
> > > `pytest --bigdata test_miri_image.py`
> > > Note that you have to first setup the environment variable `export TEST_BIGDATA=https://bytesalad.stsci.edu/artifactory` so that Jenkins knows where to get the input data files used by the tests.
> > 
> > 
> > The test failed but I wonder if all errors are due to the change in the PHOTOM step itself.
> 
> That's probably the case. I noticed that the differences start with the "cal" product and proceed downstream from there, which is exactly what you'd expect from changes in the photom correction (which is the last step before "cal" is created).

So, what should I do? Just ignore it for now? Have you solved the issue with the permission? If you show me the error, I can try to see on my end what I can do

Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index 4596833ef..67ae78eac 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -61,6 +61,11 @@ imprint
 - Updated the logging to report which imprint image is being subtracted from the
   science image. [#8041]
 
+photom
+--------
+
+- Added time-dependent correction for MIRI Imager data [#8096, spacetelescope/stdatamodels#235]
+
 pixel_replace
 -------------
 
diff --git a/jwst/photom/miri_imager.py b/jwst/photom/miri_imager.py
new file mode 100644
index 000000000..96eec891a
--- /dev/null
+++ b/jwst/photom/miri_imager.py
@@ -0,0 +1,35 @@
+import logging
+import numpy as np
+
+log = logging.getLogger(__name__)
+log.setLevel(logging.DEBUG)
+
+# Routine to find a time-dependent correction to the PHOTOM value.
+
+def time_corr_photom(param, t):
+    """
+    Short Summary
+    --------------
+    Time dependent PHOTOM function.
+
+    The model parameters are amplitude, tau, t0. t0 is the reference day 
+    from which the time-dependent parameters were derived. This function will return 
+    a correction to apply to the PHOTOM value at a given MJD.
+
+    Parameters
+    ----------
+    param : numpy array
+        Set of parameters for the PHOTOM value
+    t : int
+        Modified Julian Day (MJD) of the observation
+
+    Returns
+    -------
+    corr: float
+        The time-dependent correction to the photmjsr term.
+    """
+    
+    amplitude, tau, t0 = param["amplitude"], param["tau"], param["t0"]
+    corr = amplitude * np.exp(-(t - t0)/tau)
+
+    return corr
diff --git a/jwst/photom/photom.py b/jwst/photom/photom.py
index a2c61244d..d5d695cc7 100644
--- a/jwst/photom/photom.py
+++ b/jwst/photom/photom.py
@@ -10,6 +10,7 @@ from stdatamodels.jwst.datamodels import dqflags
 
 from .. lib.wcs_utils import get_wavelengths
 from . import miri_mrs
+from . import miri_imager
 
 log = logging.getLogger(__name__)
 log.setLevel(logging.DEBUG)
@@ -468,7 +469,32 @@ class DataSet():
                 row = find_row(ftab.phot_table, fields_to_match)
                 if row is None:
                     return
-            self.photom_io(ftab.phot_table[row])
+                self.photom_io(ftab.phot_table[row])
+                
+            # Check if reference file contains the coefficients for the time-dependent correction of the PHOTOM value
+            try:
+                ftab.getarray_noinit("timecoeff")
+                log.info("Applying the time-dependent correction to the PHOTOM value.")
+
+                mid_time = self.input.meta.exposure.mid_time
+                photom_corr = miri_imager.time_corr_photom(ftab.timecoeff[row], mid_time)
+
+                data = np.array(
+                    [(self.filter, self.subarray, ftab.phot_table[row]['photmjsr']+photom_corr, ftab.phot_table[row]['uncertainty'])],
+                    dtype=[
+                        ("filter", "O"),
+                        ("subarray", "O"),
+                        ("photmjsr", "<f4"),
+                        ("uncertainty", "<f4")
+                        ],
+                )
+                fftab = datamodels.MirImgPhotomModel(phot_table=data)
+                self.photom_io(fftab.phot_table[0])
+            except AttributeError:
+                # No time-dependent correction is applied
+                log.info(" Skipping MIRI imager time correction. Extension not found in the reference file.")
+                self.photom_io(ftab.phot_table[row])
+
         # MRS detectors
         elif self.detector == 'MIRIFUSHORT' or self.detector == 'MIRIFULONG':
 


Test patch:
diff --git a/jwst/photom/tests/test_photom.py b/jwst/photom/tests/test_photom.py
index a4a8a46b4..f10105cd8 100644
--- a/jwst/photom/tests/test_photom.py
+++ b/jwst/photom/tests/test_photom.py
@@ -340,6 +340,7 @@ def create_input(instrument, detector, exptype,
             input_model.var_flat = np.ones(shape, dtype=np.float32)
             input_model.meta.subarray.name = 'SUB256'       # matches 'GENERIC'
             input_model.meta.target.source_type = 'POINT'
+            input_model.meta.exposure.mid_time = 60000.0 # Added for new PHOTOM step
             input_model.meta.photometry.pixelarea_arcsecsq = 0.0025
             input_model.meta.photometry.pixelarea_steradians = 0.0025 * A2_TO_SR
     elif instrument == 'FGS':
@@ -780,7 +781,7 @@ def create_photom_miri_image(min_wl=16.5, max_wl=19.5,
     """
 
     filter = ["F1800W", "F2100W", "F2550W"]
-    subarray = ["GENERIC", "GENERIC", "GENERIC"]
+    subarray = ["SUB256", "SUB256", "SUB256"]
 
     nrows = len(filter)
 
@@ -794,7 +795,15 @@ def create_photom_miri_image(min_wl=16.5, max_wl=19.5,
                       ('uncertainty', '<f4')])
     reftab = np.array(list(zip(filter, subarray, photmjsr, uncertainty)),
                       dtype=dtype)
-    ftab = datamodels.MirImgPhotomModel(phot_table=reftab)
+    timecoeff_amp = np.linspace(2.1, 2.1 + (nrows - 1.) * 0.1, nrows)
+    timecoeff_tau = np.asarray([145, 145, 145])
+    timecoeff_t0 = np.asarray([59720, 59720, 59720])
+    dtypec = np.dtype([('amplitude', '<f4'),
+                      ('tau', '<f4'),
+                      ('t0', '<f4')])
+    reftabc = np.array(list(zip(timecoeff_amp, timecoeff_tau, timecoeff_t0)),
+                      dtype=dtypec)
+    ftab = datamodels.MirImgPhotomModel(phot_table=reftab, timecoeff=reftabc)
 
     return ftab
 
@@ -1415,10 +1424,13 @@ def test_miri_image():
     rownum = find_row_in_ftab(save_input, ftab, ['filter'],
                               slitname=None, order=None)
     photmjsr = ftab.phot_table['photmjsr'][rownum]
+    amplitude = ftab.timecoeff['amplitude'][rownum]
+    tau = ftab.timecoeff['tau'][rownum]
+    t0 = ftab.timecoeff['t0'][rownum]
     shape = input.shape
     ix = shape[1] // 2
     iy = shape[0] // 2
-    compare = photmjsr
+    compare = photmjsr + amplitude * np.exp(-(60000 - t0)/tau) # Added for new PHOTOM step
     # Compare the values at the center pixel.
     ratio = output[iy, ix] / input[iy, ix]
     assert np.allclose(ratio, compare, rtol=1.e-7)