Repository: spacetelescope/jwst

Issue:
Update NIRSpec fixed-slit datamodels to include flatfield and photom correction arrays
_Issue [JP-1774](https://jira.stsci.edu/browse/JP-1774) was created on JIRA by [Howard Bushouse](https://jira.stsci.edu/secure/ViewProfile.jspa?name=bushouse):_

In order to implement the NIRSpec fixed-slit master background corrections (JP-1761) the datamodel(s) used by NIRSpec fixed-slit data in calspec2 and calspec3 processing must be updated to include correction arrays for FlatField uniform source, FlatField point source, Photom uniform source, and Photom point source, in the same way that Pathloss correction arrays are already included.

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/spacetelescope/jwst/pull/5460?src=pr&el=h1) Report
> Merging [#5460](https://codecov.io/gh/spacetelescope/jwst/pull/5460?src=pr&el=desc) into [master](https://codecov.io/gh/spacetelescope/jwst/commit/95b704843fe96238b49e27b2b4e4e4db68aa5ae7?el=desc) will **increase** coverage by `0.02%`.
> The diff coverage is `88.88%`.

[![Impacted file tree graph](https://codecov.io/gh/spacetelescope/jwst/pull/5460/graphs/tree.svg?width=650&height=150&src=pr&token=Utf5Zs9g7z)](https://codecov.io/gh/spacetelescope/jwst/pull/5460?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5460      +/-   ##
==========================================
+ Coverage   71.85%   71.88%   +0.02%     
==========================================
  Files         414      414              
  Lines       37756    37822      +66     
  Branches     5844     5865      +21     
==========================================
+ Hits        27131    27189      +58     
- Misses       8909     8915       +6     
- Partials     1716     1718       +2     
```

| Flag | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| nightly | `71.85% <92.30%> (ø)` | | Carriedforward from [95b7048](https://codecov.io/gh/spacetelescope/jwst/commit/95b704843fe96238b49e27b2b4e4e4db68aa5ae7) |
| unit | `52.44% <0.00%> (-0.01%)` | :arrow_down: | |

*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags) to find out more.

| [Impacted Files](https://codecov.io/gh/spacetelescope/jwst/pull/5460?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [jwst/datamodels/multiexposure.py](https://codecov.io/gh/spacetelescope/jwst/pull/5460/diff?src=pr&el=tree#diff-andzdC9kYXRhbW9kZWxzL211bHRpZXhwb3N1cmUucHk=) | `63.76% <25.00%> (+3.39%)` | :arrow_up: |
| [jwst/datamodels/multislit.py](https://codecov.io/gh/spacetelescope/jwst/pull/5460/diff?src=pr&el=tree#diff-andzdC9kYXRhbW9kZWxzL211bHRpc2xpdC5weQ==) | `87.75% <100.00%> (+14.22%)` | :arrow_up: |
| [jwst/datamodels/slit.py](https://codecov.io/gh/spacetelescope/jwst/pull/5460/diff?src=pr&el=tree#diff-andzdC9kYXRhbW9kZWxzL3NsaXQucHk=) | `80.45% <100.00%> (-0.31%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/spacetelescope/jwst/pull/5460?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/spacetelescope/jwst/pull/5460?src=pr&el=footer). Last update [95b7048...c5dfc3c](https://codecov.io/gh/spacetelescope/jwst/pull/5460?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index e74a92e51..fef11b887 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -45,6 +45,8 @@ datamodels
 - Updated keyword schemas for EXP_TYPE and MODULE, to keep in sync with the
   JWST Keyword Dictionary [#5452]
 
+- Added flatfield and photom correction arrays to slit data models [#5460]
+
 extract_1d
 ----------
 
diff --git a/jwst/datamodels/multiexposure.py b/jwst/datamodels/multiexposure.py
index 96ec90172..bf3f614c2 100644
--- a/jwst/datamodels/multiexposure.py
+++ b/jwst/datamodels/multiexposure.py
@@ -67,8 +67,12 @@ class MultiExposureModel(DataModel):
             self.exposures[0].var_flat = init.var_flat
             self.exposures[0].wavelength = init.wavelength
             self.exposures[0].barshadow = init.barshadow
+            self.exposures[0].flatfield_point = init.flatfield_point
+            self.exposures[0].flatfield_uniform = init.flatfield_uniform
             self.exposures[0].pathloss_point = init.pathloss_point
             self.exposures[0].pathloss_uniform = init.pathloss_uniform
+            self.exposures[0].photom_point = init.photom_point
+            self.exposures[0].photom_uniform = init.photom_uniform
             self.exposures[0].area = init.area
             return
 
diff --git a/jwst/datamodels/multislit.py b/jwst/datamodels/multislit.py
index b412ecd6f..3e6203f30 100644
--- a/jwst/datamodels/multislit.py
+++ b/jwst/datamodels/multislit.py
@@ -49,12 +49,24 @@ class MultiSlitModel(model_base.DataModel):
     slits.items.barshadow : numpy float32 array
          Bar shadow correction
 
+    slits.items.flatfield_point : numpy float32 array
+         flatfield array for point source
+
+    slits.items.flatfield_uniform : numpy float32 array
+         flatfield array for uniform source
+
     slits.items.pathloss_point : numpy float32 array
          pathloss array for point source
 
     slits.items.pathloss_uniform : numpy float32 array
          pathloss array for uniform source
 
+    slits.items.photom_point : numpy float32 array
+         photom array for point source
+
+    slits.items.photom_uniform : numpy float32 array
+         photom array for uniform source
+
     slits.items.area : numpy float32 array
          Pixel area map array
     """
diff --git a/jwst/datamodels/schemas/flatcorr.schema.yaml b/jwst/datamodels/schemas/flatcorr.schema.yaml
new file mode 100644
index 000000000..916cd0a50
--- /dev/null
+++ b/jwst/datamodels/schemas/flatcorr.schema.yaml
@@ -0,0 +1,16 @@
+%YAML 1.1
+---
+$schema: "http://stsci.edu/schemas/fits-schema/fits-schema"
+id: "http://stsci.edu/schemas/jwst_datamodel/flatcorr.schema"
+type: object
+properties:
+  flatfield_point:
+    title: Flat-field point source correction
+    fits_hdu: FLATFIELD_PS
+    ndim: 2
+    datatype: float32
+  flatfield_uniform:
+    title: Flat-field uniform source correction
+    fits_hdu: FLATFIELD_UN
+    ndim: 2
+    datatype: float32
diff --git a/jwst/datamodels/schemas/pathlosscorr.schema.yaml b/jwst/datamodels/schemas/pathlosscorr.schema.yaml
index ae2c11cba..10871045c 100644
--- a/jwst/datamodels/schemas/pathlosscorr.schema.yaml
+++ b/jwst/datamodels/schemas/pathlosscorr.schema.yaml
@@ -10,7 +10,7 @@ properties:
     ndim: 2
     datatype: float32
   pathloss_uniform:
-    title: Pathloss uniformsource correction
+    title: Pathloss uniform source correction
     fits_hdu: PATHLOSS_UN
     ndim: 2
     datatype: float32
diff --git a/jwst/datamodels/schemas/photomcorr.schema.yaml b/jwst/datamodels/schemas/photomcorr.schema.yaml
new file mode 100644
index 000000000..77496a100
--- /dev/null
+++ b/jwst/datamodels/schemas/photomcorr.schema.yaml
@@ -0,0 +1,16 @@
+%YAML 1.1
+---
+$schema: "http://stsci.edu/schemas/fits-schema/fits-schema"
+id: "http://stsci.edu/schemas/jwst_datamodel/photomcorr.schema"
+type: object
+properties:
+  photom_point:
+    title: Photometric point source correction
+    fits_hdu: PHOTOM_PS
+    ndim: 2
+    datatype: float32
+  photom_uniform:
+    title: Photometric uniform source correction
+    fits_hdu: PHOTOM_UN
+    ndim: 2
+    datatype: float32
diff --git a/jwst/datamodels/schemas/slitdata.schema.yaml b/jwst/datamodels/schemas/slitdata.schema.yaml
index 60841afea..ce756734d 100644
--- a/jwst/datamodels/schemas/slitdata.schema.yaml
+++ b/jwst/datamodels/schemas/slitdata.schema.yaml
@@ -48,7 +48,9 @@ allOf:
       default: 0
       datatype: int32
 - $ref: variance.schema
+- $ref: flatcorr.schema
 - $ref: pathlosscorr.schema
+- $ref: photomcorr.schema
 - $ref: slitmeta.schema
 - $ref: bunit.schema
 - $ref: photometry.schema
diff --git a/jwst/datamodels/slit.py b/jwst/datamodels/slit.py
index 50350a842..97495839c 100644
--- a/jwst/datamodels/slit.py
+++ b/jwst/datamodels/slit.py
@@ -7,7 +7,7 @@ __all__ = ['SlitModel', 'SlitDataModel']
 
 class SlitDataModel(DataModel):
     """
-    A data model for 2D images.
+    A data model for 2D slit images.
 
     Parameters
     __________
@@ -32,12 +32,24 @@ class SlitDataModel(DataModel):
     barshadow : numpy float32 array
          Bar shadow correction
 
+    flatfield_point : numpy float32 array
+         flatfield array for point source
+
+    flatfield_uniform : numpy float32 array
+         flatfield array for uniform source
+
     pathloss_point : numpy float32 array
          pathloss array for point source
 
     pathloss_uniform : numpy float32 array
          pathloss array for uniform source
 
+    photom_point : numpy float32 array
+         photom array for point source
+
+    photom_uniform : numpy float32 array
+         photom array for uniform source
+
     area : numpy float32 array
          Pixel area map array
     """
@@ -98,12 +110,24 @@ class SlitModel(DataModel):
     barshadow : numpy float32 array
          Bar shadow correction
 
+    flatfield_point : numpy float32 array
+         flatfield array for point source
+
+    flatfield_uniform : numpy float32 array
+         flatfield array for uniform source
+
     pathloss_point : numpy float32 array
          pathloss array for point source
 
     pathloss_uniform : numpy float32 array
          pathloss array for uniform source
 
+    photom_point : numpy float32 array
+         photom array for point source
+
+    photom_uniform : numpy float32 array
+         photom array for uniform source
+
     area : numpy float32 array
          Pixel area map array
 
diff --git a/jwst/datamodels/util.py b/jwst/datamodels/util.py
index 19b6f8f58..96b752066 100644
--- a/jwst/datamodels/util.py
+++ b/jwst/datamodels/util.py
@@ -10,6 +10,8 @@ from platform import system as platform_system
 import psutil
 import traceback
 
+import asdf
+
 import numpy as np
 from astropy.io import fits
 
@@ -21,6 +23,7 @@ log = logging.getLogger(__name__)
 log.addHandler(logging.NullHandler())
 
 
+
 class NoTypeWarning(Warning):
     pass
 
@@ -112,8 +115,18 @@ def open(init=None, memmap=False, **kwargs):
             return container.ModelContainer(init, **kwargs)
 
         elif file_type == "asdf":
-            # Read the file as asdf, no need for a special class
-            return model_base.DataModel(init, **kwargs)
+            if s3_utils.is_s3_uri(init):
+                asdffile = asdf.open(s3_utils.get_object(init), **kwargs)
+            else:
+                asdffile = asdf.open(init, **kwargs)
+
+            # Detect model type, then get defined model, and call it.
+            new_class = _class_from_model_type(asdffile)
+            if new_class is None:
+                # No model class found, so return generic DataModel.
+                return model_base.DataModel(asdffile, **kwargs)
+
+            return new_class(asdffile)
 
     elif isinstance(init, tuple):
         for item in init:
@@ -195,15 +208,29 @@ def open(init=None, memmap=False, **kwargs):
     return model
 
 
-def _class_from_model_type(hdulist):
+def _class_from_model_type(init):
     """
     Get the model type from the primary header, lookup to get class
+
+    Parameter
+    ---------
+    init: AsdfFile or HDUList
+
+    Return
+    ------
+    new_class: str or None
     """
     from . import _defined_models as defined_models
 
-    if hdulist:
-        primary = hdulist[0]
-        model_type = primary.header.get('DATAMODL')
+    if init:
+        if isinstance(init, fits.hdu.hdulist.HDUList):
+            primary = init[0]
+            model_type = primary.header.get('DATAMODL')
+        elif isinstance(init, asdf.AsdfFile):
+            try:
+                model_type = init.tree['meta']['model_type']
+            except KeyError:
+                model_type = None
 
         if model_type is None:
             new_class = None


Test patch:
diff --git a/jwst/datamodels/tests/test_open.py b/jwst/datamodels/tests/test_open.py
index 43cbe3400..f2117d1cb 100644
--- a/jwst/datamodels/tests/test_open.py
+++ b/jwst/datamodels/tests/test_open.py
@@ -17,6 +17,8 @@ from jwst.datamodels import (DataModel, ModelContainer, ImageModel,
 from jwst import datamodels
 from jwst.datamodels import util
 
+import asdf
+
 # Define artificial memory size
 MEMORY = 100  # 100 bytes
 
@@ -273,3 +275,34 @@ def t_path(partial_path):
     """Construction the full path for test files"""
     test_dir = os.path.join(os.path.dirname(__file__), 'data')
     return os.path.join(test_dir, partial_path)
+
+
+@pytest.mark.parametrize("suffix", ["asdf", "fits"])
+def test_open_asdf_datamodel_class(tmpdir, suffix):
+    path = str(tmpdir.join(f"image_model.{suffix}"))
+    model = ImageModel((10, 10))
+    model.save(path)
+
+    with datamodels.open(path) as m:
+        assert isinstance(m, ImageModel)
+
+
+@pytest.mark.parametrize("suffix", ["asdf", "fits"])
+def test_open_asdf_no_datamodel_class(tmpdir, suffix):
+    path = str(tmpdir.join(f"no_model.{suffix}"))
+    # model = DataModel()
+    with DataModel() as dm:
+        dm.save(path)
+
+    with datamodels.open(path) as m:
+        assert isinstance(m, DataModel)
+
+
+def test_open_asdf(tmpdir):
+    path = str(tmpdir.join("straight_asdf.asdf"))
+    tree = {"foo": 42, "bar": 13, "seq": np.arange(100)}
+    with asdf.AsdfFile(tree) as af:
+        af.write_to(path)
+
+    with datamodels.open(path) as m:
+        assert isinstance(m, DataModel)