Repository: spacetelescope/jwst

Issue:
level3 nis_image of jw01230-o009_20240210t011656_image3_00001 crashed with KeyError: 'ra'
_Issue [JP-3541](https://jira.stsci.edu/browse/JP-3541) was created on JIRA by [Hien Tran](https://jira.stsci.edu/secure/ViewProfile.jspa?name=htran):_

ops saw a new error during reprocessing of nis_image dataset [jw01230-o009_20240210t011656_image3_00001](https://pljwdmsweb.stsci.edu/owlgui/logs/directory_view?directory_path=/ifs/archive/ops/jwst/info/owl/logs/owlmgr_jw01230-o009_20240210t011656_image3_00001_1707534171.411246&fileset_name=jw01230-o009_20240210t011656_image3_00001&dag_name=REPRO_LEVEL_3) (for l3 product jw01230-o009_t002_niriss_clearp-f480m-sub80) with b10.0, in the tweakreg step
```java
2024-02-17 00:34:17,588 - CRDS - DEBUG - Final effective context is 'jwst_1197.pmap'
2024-02-17 00:34:17,588 - CRDS - DEBUG - Computing best references locally.
2024-02-17 00:34:17,589 - CRDS - DEBUG - Bestrefs header:
{'META.EXPOSURE.READPATT [READPATT]': 'NISRAPID',
'META.EXPOSURE.TYPE [EXP_TYPE]': 'NIS_IMAGE',
'META.INSTRUMENT.DETECTOR [DETECTOR]': 'NIS',
'META.INSTRUMENT.FILTER [FILTER]': 'F480M',
'META.INSTRUMENT.LAMP_STATE [LAMP]': 'NONE',
'META.INSTRUMENT.NAME [INSTRUME]': 'NIRISS',
'META.INSTRUMENT.PUPIL [PUPIL]': 'CLEARP',
'META.OBSERVATION.DATE [DATE-OBS]': '2023-04-17',
'META.OBSERVATION.TIME [TIME-OBS]': '22:24:14.405',
'META.SUBARRAY.NAME [SUBARRAY]': 'SUB80',
'META.VISIT.CROWDED_FIELD [CROWDFLD]': 'F',
'META.VISIT.TSOVISIT [TSOVISIT]': 'F',
'META.VISIT.TYPE [VISITYPE]': 'PRIME_TARGETED_FIXED',
'REFTYPE': 'UNDEFINED'}
2024-02-17 00:34:17,589 - CRDS - DEBUG - Reference type 'abvegaoffset' defined as 'jwst_niriss_abvegaoffset_0003.asdf'
2024-02-17 00:34:17,589 - CRDS - DEBUG - Reference type 'apcorr' defined as 'jwst_niriss_apcorr_0008.fits'
2024-02-17 00:34:17,589 - CRDS - DEBUG - Reference type 'drizpars' defined as 'jwst_niriss_drizpars_0002.fits'
2024-02-17 00:34:17,658 - stpipe.Image3Pipeline - INFO - Prefetch for ABVEGAOFFSET reference file is '/ifs/archive/ops/jwst/ref/tmp_crds/crds/cache/references/jwst/niriss/jwst_niriss_abvegaoffset_0003.asdf'.
2024-02-17 00:34:17,659 - stpipe.Image3Pipeline - INFO - Prefetch for APCORR reference file is '/ifs/archive/ops/jwst/ref/tmp_crds/crds/cache/references/jwst/niriss/jwst_niriss_apcorr_0008.fits'.
2024-02-17 00:34:17,660 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/ifs/archive/ops/jwst/ref/tmp_crds/crds/cache/references/jwst/niriss/jwst_niriss_drizpars_0002.fits'.
2024-02-17 00:34:17,661 - stpipe.Image3Pipeline - INFO - Starting calwebb_image3 ...
2024-02-17 00:34:17,870 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg running with args (<ModelContainer>,).
2024-02-17 00:34:17,872 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '/ifs/archive/ops/jwst/info/owlmgr/paths/sdp/asn_creation/cal/level3', 'save_catalogs': False, 'use_custom_catalogs': False, 'catalog_format': 'ecsv', 'catfile': '', 'kernel_fwhm': 2.5, 'snr_threshold': 10, 'sharplo': 0.2, 'sharphi': 1.0, 'roundlo': -1.0, 'roundhi': 1.0, 'brightest': 100, 'peakmax': None, 'bkg_boxsize': 400, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 15, 'searchrad': 1.0, 'use2dhist': True, 'separation': 1.0, 'tolerance': 1.0, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'shift', 'nclip': 3, 'sigma': 3.0, 'abs_refcat': 'GAIADR3', 'save_abs_catalog': False, 'abs_minobj': 15, 'abs_searchrad': 6.0, 'abs_use2dhist': True, 'abs_separation': 0.1, 'abs_tolerance': 0.7, 'abs_fitgeometry': 'rshift', 'abs_nclip': 3, 'abs_sigma': 3.0}
2024-02-17 00:34:17,878 - stpipe.Image3Pipeline.tweakreg - WARNING - /dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/photutils/background/background_2d.py:274: AstropyUserWarning: Input data contains invalid values (NaNs or infs), which were automatically masked.
2024-02-17 00:34:17,879 - stpipe.Image3Pipeline.tweakreg - WARNING - warnings.warn('Input data contains invalid values (NaNs or '
2024-02-17 00:34:17,879 - stpipe.Image3Pipeline.tweakreg - WARNING -
2024-02-17 00:34:17,879 - stpipe.Image3Pipeline.tweakreg - WARNING - /dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/photutils/background/background_2d.py:274: AstropyUserWarning: Input data contains invalid values (NaNs or infs), which were automatically masked.
2024-02-17 00:34:17,879 - stpipe.Image3Pipeline.tweakreg - WARNING - warnings.warn('Input data contains invalid values (NaNs or '
2024-02-17 00:34:17,879 - stpipe.Image3Pipeline.tweakreg - WARNING -
2024-02-17 00:34:17,880 - stpipe.Image3Pipeline.tweakreg - INFO - Background could not be estimated in meshes. Using the entire unmasked array for background estimation: bkg_boxsize=(80, 80).
2024-02-17 00:34:17,886 - stpipe.Image3Pipeline.tweakreg - INFO - Detected 10 sources in jw01230009001_03103_00001_nis_cal.fits.
2024-02-17 00:34:17,894 - stpipe.Image3Pipeline.tweakreg - INFO -
2024-02-17 00:34:17,894 - stpipe.Image3Pipeline.tweakreg - INFO - Number of image groups to be aligned: 1.
2024-02-17 00:34:17,894 - stpipe.Image3Pipeline.tweakreg - INFO - Image groups:
2024-02-17 00:34:17,913 - stpipe.Image3Pipeline.tweakreg - INFO - * Images in GROUP 'jw01230009001_03103_00001_nis_cal':
2024-02-17 00:34:17,914 - stpipe.Image3Pipeline.tweakreg - INFO - jw01230009001_03103_00001_nis_cal
2024-02-17 00:34:17,914 - stpipe.Image3Pipeline.tweakreg - INFO -
Traceback (most recent call last):
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/bin/strun", line 26, in <module>
step = Step.from_cmdline(sys.argv[1:])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/stpipe/step.py", line 186, in from_cmdline
return cmdline.step_from_cmdline(args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/stpipe/cmdline.py", line 386, in step_from_cmdline
step.run(*positional)
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/stpipe/step.py", line 478, in run
step_result = self.process(*args)
^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/jwst/pipeline/calwebb_image3.py", line 89, in process
input_models = self.tweakreg(input_models)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/stpipe/step.py", line 478, in run
step_result = self.process(*args)
^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/jwst/tweakreg/tweakreg_step.py", line 392, in process
ref_cat = amutils.create_astrometric_catalog(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/jwst/tweakreg/astrometric_utils.py", line 98, in create_astrometric_catalog
ref_table = ref_dict[colnames]
~~~~~~~~^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/astropy/table/table.py", line 2064, in __getitem__
[self[x] for x in item], copy_indices=self._copy_indices
^^^^^^^^^^^^^^^^^^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/astropy/table/table.py", line 2064, in <listcomp>
[self[x] for x in item], copy_indices=self._copy_indices
~~~~^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/astropy/table/table.py", line 2055, in __getitem__
return self.columns[item]
~~~~~~~~~~~~^^^^^^
File "/dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/astropy/table/table.py", line 264, in __getitem__
return OrderedDict.__getitem__(self, item)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'ra'
2024048003418 INFO src=ssb_calibration_wrapper._strun_analyze_and_log_failure fsn=jw01230-o009_20240210t011656_image3_00001_asn msg="strun /dms/local/jwst/pipeline/pkgs/miniconda3/envs/jwstdp-1.12.5.20231019-py3.11/lib/python3.11/site-packages/jwst/pipeline/calwebb_image3.cfg FAILED (exit=1) on jw01230-o009_20240210t011656_image3_00001_asn.json."
2024048003418 ERROR src=ssb_calibration_wrapper.calibrate._strun._strun_analyze_and_log_failure fsn=jw01230-o009_20240210t011656_image3_00001_asn msg="strun exit status=1" ```
we successfully generate l3 products for this same dataset before and did not see such error in previous builds.

Comments:

Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index 5cedc0faa..1abae4c32 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -16,6 +16,15 @@ background
 - Updated to allow multi-integration (rateints) background exposures to have
   a different value of NINTS than the science exposure. [#8326]
 
+charge_migration
+----------------
+
+- Updated the CHARGELOSS flagging.  In an integration ramp, the first group in
+  the SCI data is found that is above the CHARGELOSS threshold and not flagged
+  as DO_NOT_USE.  This group, and all subsequent groups, are then flagged as
+  CHARGELOSS and DO_NOT_USE.  The four nearest pixel neighbor are then flagged
+  in the same group. [#8336]
+
 cube_build
 ----------
 
@@ -51,6 +60,12 @@ emicorr
 - Set skip=True by default in the code, to be turned on later by a parameter
   reference file. [#8171]
 
+exp_to_source
+-------------
+
+- Fixed a bug for multislit data that bunit values, model_type and wcsinfo was
+  was being overwritten with the top multispec model values. [#8294]
+
 extract_1d
 ----------
 
@@ -88,6 +103,10 @@ general
 
 - Update minimum required stdatamodels version to include 1.10.0 [#8322]
 
+- Update minimum required gwcs version to include 0.21.0 [#8337]
+
+- Remove unused asdf-transform-schemas dependency [#8337]
+
 jump
 ----
 
@@ -100,6 +119,13 @@ lib
 - Updated ``set_velocity_aberration`` to use datamodels instead of `astropy.io.fits` for opening
   and manipulating input files. [#8285]
 
+lib
+---
+
+- Added new function set_nans_to_donotuse in ``lib.basic_utils`` to
+  check the science data array for NaN values and check if they have
+  a DQ flag of DO_NOT_USE, or set it if not. [#8292]
+
 outlier_detection
 -----------------
 
@@ -111,13 +137,11 @@ outlier_detection
   original input files to accidentally get deleted instead of just the intermediate
   files. [#8263]
 
-resample
+pathloss
 --------
-- Updated exposure time weighting to use the measurement time 
-  (TMEASURE) when available. [#8212]
 
-- Removed product exposure time (``TEXPTIME``) from all computations
-  in the resample step. [#8212]
+- Added a check to find all NaN values in the data with a corresponding
+  even value flag in the DQ array, and convert them to DO_NOT_USE. [#8292]
 
 photom
 ------
@@ -167,6 +191,12 @@ refpix
 resample
 --------
 
+- Updated exposure time weighting to use the measurement time 
+  (TMEASURE) when available. [#8212]
+
+- Removed product exposure time (``TEXPTIME``) from all computations
+  in the resample step. [#8212]
+
 - Use the same ``iscale`` value for resampling science data and variance arrays. [#8159]
 
 - Changed to use the high-level APE 14 API (``pixel_to_world_values`` and
@@ -206,6 +236,9 @@ tweakreg
 - Suppress warnings from ``photutils.background.Background2D`` regarding
   NaNs in the input data. [#8308]
 
+- Fixed a bug that caused failures instead of warnings when no GAIA sources
+  were found within the bounding box of the input image. [#8334]
+
 
 1.13.4 (2024-01-25)
 ===================
diff --git a/jwst/charge_migration/charge_migration.py b/jwst/charge_migration/charge_migration.py
index 30e1b9bc1..ccfcc3836 100644
--- a/jwst/charge_migration/charge_migration.py
+++ b/jwst/charge_migration/charge_migration.py
@@ -5,6 +5,7 @@ import numpy as np
 
 from stdatamodels.jwst.datamodels import dqflags
 
+
 log = logging.getLogger(__name__)
 log.setLevel(logging.DEBUG)
 
@@ -72,72 +73,29 @@ def flag_pixels(data, gdq, signal_threshold):
         updated group dq array
     """
     n_ints, n_grps, n_rows, n_cols = gdq.shape
-
-    new_gdq = gdq.copy()   # Updated gdq
-
-    # Flag all exceedances with CHARGELOSS and NO_NOT_USE
-    chargeloss_pix = (data > signal_threshold) & (gdq != DNU)
-    new_gdq[chargeloss_pix] = np.bitwise_or(new_gdq[chargeloss_pix], CHLO | DNU)
-
-    # Reset groups previously flagged as DNU
-    gdq_orig = gdq.copy()  # For resetting to previously flagged DNU
-    wh_gdq_DNU = np.bitwise_and(gdq_orig, DNU)
-
-    # Get indices for exceedances
-    arg_where = np.argwhere(new_gdq == CHLO_DNU)
-
-    a_int = arg_where[:, 0]  # array of integrations
-    a_grp = arg_where[:, 1]  # array of groups
-    a_row = arg_where[:, 2]  # array of rows
-    a_col = arg_where[:, 3]  # array of columns
-
-    # Process the 4 nearest neighbors of each exceedance
-    # Pixel to the east
-    xx_max_p1 = a_col[a_col < (n_cols-1)] + 1
-    i_int = a_int[a_col < (n_cols-1)]
-    i_grp = a_grp[a_col < (n_cols-1)]
-    i_row = a_row[a_col < (n_cols-1)]
-
-    if len(xx_max_p1) > 0:
-        new_gdq[i_int, i_grp, i_row, xx_max_p1] = \
-            np.bitwise_or(new_gdq[i_int, i_grp, i_row, xx_max_p1], CHLO | DNU)
-
-    new_gdq[wh_gdq_DNU == 1] = gdq_orig[wh_gdq_DNU == 1]  # reset for earlier DNUs
-
-    # Pixel to the west
-    xx_m1 = a_col[a_col > 0] - 1
-    i_int = a_int[a_col > 0]
-    i_grp = a_grp[a_col > 0]
-    i_row = a_row[a_col > 0]
-
-    if len(xx_m1) > 0:
-        new_gdq[i_int, i_grp, i_row, xx_m1] = \
-            np.bitwise_or(new_gdq[i_int, i_grp, i_row, xx_m1], CHLO | DNU)
-
-    new_gdq[wh_gdq_DNU == 1] = gdq_orig[wh_gdq_DNU == 1]  # reset for earlier DNUs
-
-    # Pixel to the north
-    yy_m1 = a_row[a_row > 0] - 1
-    i_int = a_int[a_row > 0]
-    i_grp = a_grp[a_row > 0]
-    i_col = a_col[a_row > 0]
-
-    if len(yy_m1) > 0:
-        new_gdq[i_int, i_grp, yy_m1, i_col] = \
-            np.bitwise_or(new_gdq[i_int, i_grp, yy_m1, i_col], CHLO | DNU)
-
-    new_gdq[wh_gdq_DNU == 1] = gdq_orig[wh_gdq_DNU == 1]  # reset for earlier DNUs
-
-    # Pixel to the south
-    yy_max_p1 = a_row[a_row < (n_rows-1)] + 1
-    i_int = a_int[a_row < (n_rows-1)]
-    i_grp = a_grp[a_row < (n_rows-1)]
-    i_col = a_col[a_row < (n_rows-1)]
-
-    if len(yy_max_p1) > 0:
-        new_gdq[i_int, i_grp, yy_max_p1, i_col] = \
-            np.bitwise_or(new_gdq[i_int, i_grp, yy_max_p1, i_col], CHLO | DNU)
-
-    new_gdq[wh_gdq_DNU == 1] = gdq_orig[wh_gdq_DNU == 1]  # reset for earlier DNUs
+    chargeloss_pix = np.where((data > signal_threshold) & (gdq != DNU))
+
+    new_gdq = gdq.copy()
+    
+    for k in range(len(chargeloss_pix[0])): 
+        integ, group = chargeloss_pix[0][k], chargeloss_pix[1][k]
+        row, col = chargeloss_pix[2][k], chargeloss_pix[3][k]
+        new_gdq[integ, group:, row, col] |= CHLO_DNU
+
+        # North
+        if row > 0:
+            new_gdq[integ, group:, row-1, col] |= CHLO_DNU
+
+        # South
+        if row < (n_rows-1):
+            new_gdq[integ, group:, row+1, col] |= CHLO_DNU
+
+        # East
+        if col < (n_cols-1):
+            new_gdq[integ, group:, row, col+1] |= CHLO_DNU
+
+        # West
+        if col > 0:
+            new_gdq[integ, group:, row, col-1] |= CHLO_DNU
 
     return new_gdq
diff --git a/jwst/exp_to_source/exp_to_source.py b/jwst/exp_to_source/exp_to_source.py
index c8765cd21..b9540b1b1 100644
--- a/jwst/exp_to_source/exp_to_source.py
+++ b/jwst/exp_to_source/exp_to_source.py
@@ -40,8 +40,26 @@ def exp_to_source(inputs):
             log.debug(f'Copying source {slit.source_id}')
             result_slit = result[str(slit.source_id)]
             result_slit.exposures.append(slit)
+            # store values for later use (after merge_tree)
+            # these values are incorrectly getting overwritten by
+            # the top model.
+            slit_bunit = slit.meta.bunit_data
+            slit_bunit_err = slit.meta.bunit_err
+            slit_model = slit.meta.model_type
+            slit_wcsinfo = slit.meta.wcsinfo.instance
+            # exposure.meta.bunit_data and bunit_err does not exist
+            # before calling merge_tree save these values
+            # Before merge_tree the slits have a model_type of SlitModel.
+            # After merge_tree it is overwritten with MultiSlitModel.
+            # store the model type to undo overwriting of modeltype.
+
             merge_tree(result_slit.exposures[-1].meta.instance, exposure.meta.instance)
 
+            result_slit.exposures[-1].meta.bunit_data = slit_bunit
+            result_slit.exposures[-1].meta.bunit_err = slit_bunit_err
+            result_slit.exposures[-1].meta.model_type = slit_model
+            result_slit.exposures[-1].meta.wcsinfo = slit_wcsinfo
+
             if result_slit.meta.instrument.name is None:
                 result_slit.update(exposure)
 
diff --git a/jwst/lib/basic_utils.py b/jwst/lib/basic_utils.py
index 28971ddb8..92e77dcf4 100644
--- a/jwst/lib/basic_utils.py
+++ b/jwst/lib/basic_utils.py
@@ -1,5 +1,31 @@
 """General utility objects"""
 
+from stdatamodels.jwst.datamodels import dqflags
+import numpy as np
+
+
+def set_nans_to_donotuse(data, dq):
+    """Set all NaN values in the data that have an even value to
+    DO_NOT_USE.
+
+    Parameters
+    ----------
+    data : numpy array
+        The science data array to find NaN values and
+        check of these have a DQ flag=DO_NOT_USE, or
+        set it if not.
+
+    dq : numpy array
+        The DQ array to be checked.
+
+    Returns
+    -------
+    dq : numpy array
+        The updated DQ array.
+    """
+    dq[np.isnan(data)] |= dqflags.pixel['DO_NOT_USE']
+    return dq
+
 
 class LoggingContext:
     """Logging context manager
diff --git a/jwst/pathloss/pathloss.py b/jwst/pathloss/pathloss.py
index c3339bd27..4ad2f551b 100644
--- a/jwst/pathloss/pathloss.py
+++ b/jwst/pathloss/pathloss.py
@@ -10,6 +10,8 @@ import stdatamodels.jwst.datamodels as datamodels
 
 from jwst.assign_wcs import nirspec, util
 from jwst.lib.wcs_utils import get_wavelengths
+from jwst.lib.basic_utils import set_nans_to_donotuse
+
 
 log = logging.getLogger(__name__)
 log.setLevel(logging.DEBUG)
@@ -545,6 +547,9 @@ def do_correction_mos(data, pathloss, inverse=False, source_type=None, correctio
         slit.pathloss_point = correction.pathloss_point
         slit.pathloss_uniform = correction.pathloss_uniform
 
+        # check the dq flags have the correct value
+        slit.dq = set_nans_to_donotuse(slit.data, slit.dq)
+
     # Set step status to complete
     data.meta.cal_step.pathloss = 'COMPLETE'
 
@@ -608,6 +613,9 @@ def do_correction_fixedslit(data, pathloss, inverse=False, source_type=None, cor
         slit.pathloss_point = correction.pathloss_point
         slit.pathloss_uniform = correction.pathloss_uniform
 
+        # check the dq flags have the correct value
+        slit.dq = set_nans_to_donotuse(slit.data, slit.dq)
+
     # Set step status to complete
     data.meta.cal_step.pathloss = 'COMPLETE'
 
@@ -661,6 +669,9 @@ def do_correction_ifu(data, pathloss, inverse=False, source_type=None, correctio
     # This might be useful to other steps
     data.wavelength = correction.wavelength
 
+    # check the dq flags have the correct value
+    data.dq = set_nans_to_donotuse(data.data, data.dq)
+
     # Set the step status to complete
     data.meta.cal_step.pathloss = 'COMPLETE'
 
@@ -702,6 +713,9 @@ def do_correction_lrs(data, pathloss, user_slit_loc):
     # This might be useful to other steps
     data.wavelength = correction.wavelength
 
+    # check the dq flags have the correct value
+    data.dq = set_nans_to_donotuse(data.data, data.dq)
+
     # Set the step status to complete
     data.meta.cal_step.pathloss = 'COMPLETE'
 
@@ -794,6 +808,9 @@ def do_correction_soss(data, pathloss):
         data.var_flat /= pathloss_2d**2
     data.pathloss_point = pathloss_2d
 
+    # check the dq flags have the correct value
+    data.dq = set_nans_to_donotuse(data.data, data.dq)
+
     # Set step status to complete
     data.meta.cal_step.pathloss = 'COMPLETE'
 
diff --git a/jwst/tweakreg/astrometric_utils.py b/jwst/tweakreg/astrometric_utils.py
index 0144fdbb7..a3034cb6c 100644
--- a/jwst/tweakreg/astrometric_utils.py
+++ b/jwst/tweakreg/astrometric_utils.py
@@ -96,8 +96,10 @@ def create_astrometric_catalog(input_models, catalog="GAIADR3", output="ref_cat.
         else Time(input_models[0].meta.observation.date).decimalyear
     )
     ref_dict = get_catalog(fiducial[0], fiducial[1], epoch=epoch, sr=radius, catalog=catalog)
+    if len(ref_dict) == 0:
+        return ref_dict
+    
     colnames = ('ra', 'dec', 'mag', 'objID', 'epoch')
-
     ref_table = ref_dict[colnames]
 
     # Add catalog name as meta data
diff --git a/pyproject.toml b/pyproject.toml
index 20dbc5433..0bbbdfa71 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -20,12 +20,11 @@ classifiers = [
 ]
 dependencies = [
     "asdf>=2.15.1,<4",
-    "asdf-transform-schemas>=0.3.0",
     "astropy>=5.3",
     "BayesicFitting>=3.0.1",
     "crds>=11.17.14",
     "drizzle>=1.14.3,<1.15.0",
-    "gwcs>=0.20.0,<0.21.0",
+    "gwcs>=0.21.0,<0.22.0",
     "numpy>=1.22",
     "opencv-python-headless>=4.6.0.66",
     "photutils>=1.5.0",


Test patch:
diff --git a/jwst/charge_migration/tests/test_charge_migration.py b/jwst/charge_migration/tests/test_charge_migration.py
index 4a2e53f73..bc6d390c5 100644
--- a/jwst/charge_migration/tests/test_charge_migration.py
+++ b/jwst/charge_migration/tests/test_charge_migration.py
@@ -8,6 +8,7 @@ from jwst.charge_migration.charge_migration_step import ChargeMigrationStep
 
 import numpy.testing as npt
 
+
 test_dq_flags = dqflags.pixel
 GOOD = test_dq_flags["GOOD"]
 DNU = test_dq_flags["DO_NOT_USE"]
@@ -78,6 +79,7 @@ def test_pix_1():
     true_out_gdq[0, 4:, 0, 0] = CHLO_DNU
 
     out_model = charge_migration(ramp_model, signal_threshold)
+
     out_data = out_model.data
     out_gdq = out_model.groupdq
 
@@ -85,113 +87,154 @@ def test_pix_1():
     npt.assert_array_equal(out_gdq, true_out_gdq)
 
 
-def test_too_few_groups():
+def test_pix_2():
     """
-    Test that processing for datasets having too few (<3) groups per integration
-    are skipped.
+    Test a later group being below the threshold.
     """
-    ngroups, nints, nrows, ncols = 2, 1, 1, 1
+    ngroups, nints, nrows, ncols = 10, 1, 1, 1
     ramp_model, pixdq, groupdq, err = create_mod_arrays(
         ngroups, nints, nrows, ncols)
 
-    ramp_model.data[0, :, 0, 0] = 20000.
-    sig_thresh = 100.
+    signal_threshold = 4000.
 
-    result = ChargeMigrationStep.call(ramp_model, skip=False,
-                                              signal_threshold=sig_thresh)
-    status = result.meta.cal_step.charge_migration
+    arr = [1000., 2000., 4005., 4500., 5000., 5500., 3500., 6000., 6500., 3700.]
+    ramp_model.data[0, :, 0, 0] = np.array(arr, dtype=np.float32)
+    arr = [0, DNU, 0, 0, 0,  0, 0, 0, 0, 0]
+    ramp_model.groupdq[0, :, 0, 0] = np.array(arr, dtype=np.uint8)
 
-    npt.assert_string_equal(status, "SKIPPED")
+    out_model = charge_migration(ramp_model, signal_threshold)
 
+    truth_arr = [0, DNU, CHLO_DNU, CHLO_DNU, CHLO_DNU, CHLO_DNU, CHLO_DNU, CHLO_DNU, CHLO_DNU, CHLO_DNU]
+    truth_gdq = np.array(truth_arr, dtype=np.uint8)
 
-def test_flag_neighbors():
+    npt.assert_array_equal(truth_gdq, out_model.groupdq[0, :, 0, 0])
+
+
+
+def nearest_neighbor_base(chg_thresh, pixel):
     """
-    Test flagging of 4 nearest neighbors of exceedances. Tests pixels on
-    array edges, Tests exclusion of groups previously flagged as DO_NOT_USE.
+    Set up ramp array that is 5, 5 with 10 groups.
+    The flagging starts in group 3 (zero based) in the pixel tested.
     """
-    ngroups, nints, nrows, ncols = 6, 1, 4, 3
+    nints, ngroups, nrows, ncols = 1, 10, 5, 5
     ramp_model, pixdq, groupdq, err = create_mod_arrays(
         ngroups, nints, nrows, ncols)
 
-    signal_threshold = 4400.
+    # Set up dummy data
+    base = chg_thresh * 0.05
+    base_arr = [float(k+1) * base for k in range(ngroups)]
+    for row in range(nrows):
+        for col in range(ncols):
+            ramp_model.data[0, :, row, col] = np.array(base_arr, dtype=np.float32)
 
-    # Populate pixel-specific SCI and GROUPDQ arrays
-    ramp_model.data[0, :, :, :] = \
-        np.array([[
-            [1900., 2666., 2100.],
-            [3865., 2300., 3177.],
-            [3832., 3044., 3588.],
-            [3799., 3233., 3000.]],
-
-           [[2100., 2866., 2300.],
-            [4065., 2500., 3377.],
-            [4032., 3244., 3788.],
-            [3999., 3433., 3200.]],
-
-           [[2300., 3066., 2500.],
-            [4265., 2700., 3577.],
-            [4232., 3444., 3988.],
-            [4199., 3633., 3400.]],
-
-           [[2500., 3266., 2700.],
-            [4465., 2900., 3777.],
-            [4432., 3644., 4188.],
-            [4399., 3833., 3600.]],
-
-           [[2700., 3466., 2900.],
-            [4665., 3100., 3977.],
-            [4632., 3844., 4388.],
-            [4599., 4033., 3800.]],
-
-           [[2900., 3666., 3100.],
-            [4865., 3300., 4177.],
-            [4832., 4044., 4588.],
-            [4799., 4233., 4000.]]], dtype=np.float32)
-
-    # These group DQ values should propagate unchanged to the output
-    ramp_model.groupdq[:, 4, 2, 0] = [DNU]
-    ramp_model.groupdq[:, 1, 2, 2] = [DNU]
-    ramp_model.groupdq[:, 2, 1, 1] = [DROU + DNU]
+    # Make CHARGELOSS threshold starting at group 3
+    in_row, in_col = pixel
+    ramp_model.data[0, 3:, in_row, in_col] += chg_thresh
 
-    out_model = charge_migration(ramp_model, signal_threshold)
+    return ramp_model, pixdq, groupdq, err
 
-    out_gdq = out_model.groupdq
 
-    true_out_gdq = ramp_model.groupdq.copy()
-    true_out_gdq[0, :, :, :] = \
-        np.array([[
-            [0,   0,   0],
-            [0,   0,   0],
-            [0,   0,   0],
-            [0,   0,   0]],
-
-           [[0,   0,   0],
-            [0,   0,   0],
-            [0,   0,   DNU],
-            [0,   0,   0]],
-
-           [[0,   0,   0],
-            [0,   9,   0],
-            [0,   0,   0],
-            [0,   0,   0]],
-
-           [[CHLO_DNU,   0,      0],
-            [CHLO_DNU, CHLO_DNU, 0],
-            [CHLO_DNU, CHLO_DNU, 0],
-            [CHLO_DNU,   0,      0]],
-
-           [[CHLO_DNU,   0,      0],
-            [CHLO_DNU, CHLO_DNU, 0],
-            [DNU,        0,      0],
-            [CHLO_DNU, CHLO_DNU, 0]],
-
-           [[CHLO_DNU,   0,       0],
-            [CHLO_DNU, CHLO_DNU, CHLO_DNU],
-            [CHLO_DNU, CHLO_DNU, CHLO_DNU],
-            [CHLO_DNU, CHLO_DNU, CHLO_DNU]]], dtype=np.uint8)
+def test_nearest_neighbor_1():
+    """
+    CHARGELOSS center
+    The flagging starts in group 3 (zero based) in the pixel tested.
+    """
+    chg_thresh = 4000.
+    pixel = (2, 2)
+    ramp_model, pixdq, groupdq, err = nearest_neighbor_base(chg_thresh, pixel)
+    gdq_check = ramp_model.groupdq.copy()
+    ngroups = gdq_check.shape[1]
+
+    out_model = charge_migration(ramp_model, chg_thresh)
+
+    check_pattern = [
+        [GOOD, GOOD,     GOOD,     GOOD,     GOOD],
+        [GOOD, GOOD,     CHLO_DNU, GOOD,     GOOD],
+        [GOOD, CHLO_DNU, CHLO_DNU, CHLO_DNU, GOOD],
+        [GOOD, GOOD,     CHLO_DNU, GOOD,     GOOD],
+        [GOOD, GOOD,     GOOD,     GOOD,     GOOD],
+    ]
+    check = np.array(check_pattern, dtype=gdq_check.dtype)
+    for group in range(3, ngroups):
+        gdq_check[0, group, :, :] = check
 
     npt.assert_array_equal(out_model.data, ramp_model.data)
-    npt.assert_array_equal(out_gdq, true_out_gdq)
+    npt.assert_array_equal(out_model.groupdq, gdq_check)
+
+
+def test_nearest_neighbor_2():
+    """
+    CHARGELOSS corner
+    The flagging starts in group 3 (zero based) in the pixel tested.
+    """
+    chg_thresh = 4000.
+    pixel = (0, 0)
+    ramp_model, pixdq, groupdq, err = nearest_neighbor_base(chg_thresh, pixel)
+    gdq_check = ramp_model.groupdq.copy()
+    ngroups = gdq_check.shape[1]
+
+    out_model = charge_migration(ramp_model, chg_thresh)
+
+    check_pattern = [
+        [CHLO_DNU, CHLO_DNU, GOOD, GOOD, GOOD],
+        [CHLO_DNU, GOOD,     GOOD, GOOD, GOOD],
+        [GOOD,     GOOD,     GOOD, GOOD, GOOD],
+        [GOOD,     GOOD,     GOOD, GOOD, GOOD],
+        [GOOD,     GOOD,     GOOD, GOOD, GOOD],
+    ]
+    check = np.array(check_pattern, dtype=gdq_check.dtype)
+    for group in range(3, ngroups):
+        gdq_check[0, group, :, :] = check
+
+    npt.assert_array_equal(out_model.data, ramp_model.data)
+    npt.assert_array_equal(out_model.groupdq, gdq_check)
+
+
+def test_nearest_neighbor_3():
+    """
+    CHARGELOSS Edge
+    The flagging starts in group 3 (zero based) in the pixel tested.
+    """
+    chg_thresh = 4000.
+    pixel = (2, 4)
+    ramp_model, pixdq, groupdq, err = nearest_neighbor_base(chg_thresh, pixel)
+    gdq_check = ramp_model.groupdq.copy()
+    ngroups = gdq_check.shape[1]
+
+    out_model = charge_migration(ramp_model, chg_thresh)
+
+    check_pattern = [
+        [GOOD, GOOD, GOOD, GOOD,     GOOD],
+        [GOOD, GOOD, GOOD, GOOD,     CHLO_DNU],
+        [GOOD, GOOD, GOOD, CHLO_DNU, CHLO_DNU],
+        [GOOD, GOOD, GOOD, GOOD,     CHLO_DNU],
+        [GOOD, GOOD, GOOD, GOOD,     GOOD],
+    ]
+    check = np.array(check_pattern, dtype=gdq_check.dtype)
+    for group in range(3, ngroups):
+        gdq_check[0, group, :, :] = check
+
+    npt.assert_array_equal(out_model.data, ramp_model.data)
+    npt.assert_array_equal(out_model.groupdq, gdq_check)
+
+
+def test_too_few_groups():
+    """
+    Test that processing for datasets having too few (<3) groups per integration
+    are skipped.
+    """
+    ngroups, nints, nrows, ncols = 2, 1, 1, 1
+    ramp_model, pixdq, groupdq, err = create_mod_arrays(
+        ngroups, nints, nrows, ncols)
+
+    ramp_model.data[0, :, 0, 0] = 20000.
+    sig_thresh = 100.
+
+    result = ChargeMigrationStep.call(ramp_model, skip=False,
+                                              signal_threshold=sig_thresh)
+    status = result.meta.cal_step.charge_migration
+
+    npt.assert_string_equal(status, "SKIPPED")
 
 
 def create_mod_arrays(ngroups, nints, nrows, ncols):
diff --git a/jwst/tweakreg/tests/test_amutils.py b/jwst/tweakreg/tests/test_amutils.py
index c31975f29..3f2e40fca 100644
--- a/jwst/tweakreg/tests/test_amutils.py
+++ b/jwst/tweakreg/tests/test_amutils.py
@@ -54,3 +54,22 @@ def test_create_catalog(wcsobj):
     )
     # check that we got expected number of sources
     assert len(gcat) == EXPECTED_NUM_SOURCES
+
+
+def test_create_catalog_graceful_failure(wcsobj):
+    '''
+    Ensure catalog retuns zero sources instead of failing outright
+    when the bounding box is too small to find any sources
+    '''
+    wcsobj.bounding_box = ((0, 0.5), (0, 0.5))
+
+    # Create catalog
+    gcat = amutils.create_astrometric_catalog(
+        None,
+        existing_wcs=wcsobj,
+        catalog=TEST_CATALOG,
+        output=None,
+        epoch='2016.0',
+    )
+    # check that we got expected number of sources
+    assert len(gcat) == 0