Repository: NOAA-EMC/global-workflow
Pull request #3642: Refactor marine analysis Task class
Author: login: DavidNew-NOAA
Labels: name: CI-Wcoss2-Running
description: CI testing on WCOSS for this PR is in-progress
name: CI-Gaeac6-Passed
description: **Bot use only** CI testing on Gaea C6 for this PR has completed successfully
Created: 2025-05-02T19:56:33Z | Merged: 2025-05-23T12:48:08Z

# Description

This PR refactors the marine analysis task class to be in line with the standard code structure of other JEDI domains (atmosphere, aerosol, snow). This includes:
1. Using the `JEDI` class infrastructure to initialize and run all JEDI applications used in the marine analysis task.
2. Ensuring that the input YAML for all JEDI applications are rendered with JCB.
3. Using `task_config` to render the JCB base YAML, rather than a custom dictionary.
4. Ensuring that the `task_config` member dictionary of the marine analysis `Task` class is only modified at initialization by the class constructor.
5. Ensuring that all file staging and finalizing is carried out with YAML files via the wxflow FileHandler class.

Additionally, a method has been added to the `Jedi` class which goes through the YAML it generates in the `initialize()` method and removes any observations with missing input files. This was previously done manually in the marine analysis code, but I've absorbed it into the `Jedi` class because it's useful/required for all domains.

Resolves #2931 

# Type of change
- [ ] Bug fix (fixes something broken)
- [ ] New feature (adds functionality)
- [X] Maintenance (code refactor, clean-up, new CI test, etc.)

# Change characteristics
<!-- Choose YES or NO from each of the following and delete the other -->
- Is this a breaking change (a change in existing functionality)? NO
- Does this change require a documentation update? NO
- Does this change require an update to any of the following submodules? YES 
  - [ ] EMC verif-global <!-- NOAA-EMC/EMC_verif-global#1234 -->
  - [X] GDAS NOAA-EMC/GDASApp#1656
  - [ ] GFS-utils <!-- NOAA-EMC/gfs-utils#1234 -->
  - [ ] GSI <!-- NOAA-EMC/GSI#1234 -->
  - [ ] GSI-monitor <!-- NOAA-EMC/GSI-Monitor#1234 -->
  - [ ] GSI-utils <!-- NOAA-EMC/GSI-Utils#1234 -->
  - [ ] UFS-utils <!-- ufs-community/UFS_UTILS#1234 -->
  - [ ] UFS-weather-model <!-- ufs-community/ufs-weather-model#1234 -->
  - [ ] wxflow <!-- NOAA-EMC/wxflow#1234 -->

# How has this been tested?
- Clone and build on Hera
- Run C48mx500_3DVarAOWCDA

# Checklist
- [ ] Any dependent changes have been merged and published
- [X] My code follows the style guidelines of this project
- [X] I have performed a self-review of my own code
- [X] I have commented my code, particularly in hard-to-understand areas
- [ ] I have documented my code, including function, input, and output descriptions
- [X] My changes generate no new warnings
- [X] New and existing tests pass with my changes
- [X] This change is covered by an existing CI test or a new one has been added
- [ ] Any new scripts have been added to the .github/CODEOWNERS file with owners
- [ ] I have made corresponding changes to the system documentation if necessary


Linked issue #2931: Incorporate JEDI class into marine analysis job
### What new functionality do you need?

Incorporate JEDI class into marine analysis job.

### What are the requirements for the new functionality?

For every JEDI application that is run in the marine analysis job, a JEDI object should be instantiated and utilized to initialize and run the application.

### Acceptance Criteria

Existing CI tests for the marine analysis should pass.

### Suggest a solution (optional)

_No response_
Linked issue comments: user: login: DavidNew-NOAA
created_at: 2024-09-17T14:38:52Z
body: I don't have the ability to self-assign this issue, but I plan to create a PR for this in a few weeks.
author_association: CONTRIBUTOR

Diff:
diff --git a/dev/parm/config/gfs/config.marineanl.j2 b/dev/parm/config/gfs/config.marineanl.j2
index 36861fe6d24..bc16ff509ec 100644
--- a/dev/parm/config/gfs/config.marineanl.j2
+++ b/dev/parm/config/gfs/config.marineanl.j2
@@ -5,7 +5,8 @@
 
 echo "BEGIN: config.marineanl"
 
-export JEDI_CONFIG_YAML="${PARMgfs}/gdas/soca_bmat_jedi_config.yaml.j2"
+export JEDI_CONFIG_YAML_BMAT="${PARMgfs}/gdas/soca_bmat_jedi_config.yaml.j2"
+export JEDI_CONFIG_YAML_ANALYSIS="${PARMgfs}/gdas/soca_analysis_jedi_config.yaml.j2"
 export JCB_ALGO_YAML_VAR="{{ JCB_ALGO_YAML_VAR }}"
 
 export MARINE_OBS_YAML_DIR="${PARMgfs}/gdas/soca/obs/config"
@@ -19,6 +20,8 @@ export SOCA_FIX_YAML_TMPL="${PARMgfs}/gdas/soca/soca_fix_stage_${OCNRES}.yaml.j2
 export MARINE_UTILITY_YAML_TMPL="${PARMgfs}/gdas/soca/soca_utils_stage.yaml.j2"
 export MARINE_ENSDA_STAGE_BKG_YAML_TMPL="${PARMgfs}/gdas/soca/ensda/stage_ens_mem.yaml.j2"
 export MARINE_DET_STAGE_BKG_YAML_TMPL="${PARMgfs}/gdas/soca/soca_det_bkg_stage.yaml.j2"
+export MARINE_DET_STAGE_FILES_YAML_TMPL="${PARMgfs}/gdas/soca/soca_det_files_stage.yaml.j2"
+export MARINE_DET_FINALIZE_YAML_TMPL="${PARMgfs}/gdas/soca/soca_det_finalize.yaml.j2"
 export MARINE_JCB_GDAS_ALGO="${PARMgfs}/gdas/jcb-gdas/algorithm/marine"
 export MARINE_JCB_GDAS_OBS="${PARMgfs}/gdas/jcb-gdas/observations/marine"
 
diff --git a/dev/parm/config/gfs/yaml/defaults.yaml b/dev/parm/config/gfs/yaml/defaults.yaml
index 5923b322716..7d33fcdd6e1 100644
--- a/dev/parm/config/gfs/yaml/defaults.yaml
+++ b/dev/parm/config/gfs/yaml/defaults.yaml
@@ -59,7 +59,7 @@ marineanl:
   SOCA_ANL_GEOM: "${FIXgfs}/gdas/soca/72x35x25/soca"
   SOCA_OBS_LIST: "${PARMgfs}/gdas/soca/obs/obs_list.yaml.j2"  # TODO: This is also repeated in oceanprepobs
   SOCA_NINNER: 100
-  JCB_ALGO_YAML_VAR: "${PARMgfs}/gdas/soca/marine-jcb-3dfgat.yaml.j2"
+  JCB_ALGO_YAML_VAR: "${PARMgfs}/gdas/soca/jcb-prototype_3dfgat.yaml.j2"
 
 prepoceanobs:
   SOCA_INPUT_FIX_DIR: "${FIXgfs}/gdas/soca/72x35x25/soca"
diff --git a/env/GAEAC5.env b/env/GAEAC5.env
index e81e26ed52c..ab27864848a 100755
--- a/env/GAEAC5.env
+++ b/env/GAEAC5.env
@@ -145,6 +145,8 @@ case ${step} in
 
     export NTHREADS_OCNANAL=${NTHREADSmax}
     export APRUN_MARINEANLCHKPT="${APRUN_default} --cpus-per-task=${NTHREADS_OCNANAL}"
+
+    export APRUN_MARINEANLOBSSTATS="${launcher} -n 1"
  ;;
  "marineanlletkf")
 
diff --git a/env/GAEAC6.env b/env/GAEAC6.env
index 17b633d2729..3f216e450d9 100755
--- a/env/GAEAC6.env
+++ b/env/GAEAC6.env
@@ -145,6 +145,8 @@ case ${step} in
 
     export NTHREADS_OCNANAL=${NTHREADSmax}
     export APRUN_MARINEANLCHKPT="${APRUN_default} --cpus-per-task=${NTHREADS_OCNANAL}"
+
+    export APRUN_MARINEANLOBSSTATS="${launcher} -n 1"
  ;;
  "anlstat")
 
diff --git a/env/HERA.env b/env/HERA.env
index e9fc5de4ee9..775b32d3aa9 100755
--- a/env/HERA.env
+++ b/env/HERA.env
@@ -152,6 +152,8 @@ elif [[ "${step}" = "marineanlchkpt" ]]; then
     export APRUNCFP="${launcher} -n \$ncmd --multi-prog"
     export APRUN_MARINEANLCHKPT="${APRUN_default}"
 
+    export APRUN_MARINEANLOBSSTATS="${launcher} -n 1"
+
 elif [[ "${step}" = "ocnanalecen" ]]; then
 
     export NTHREADS_OCNANALECEN=${NTHREADSmax}
diff --git a/env/HERCULES.env b/env/HERCULES.env
index 10bd5fa0068..3d42fe6fd51 100755
--- a/env/HERCULES.env
+++ b/env/HERCULES.env
@@ -151,6 +151,8 @@ case ${step} in
 
     export NTHREADS_OCNANAL=${NTHREADSmax}
     export APRUN_MARINEANLCHKPT="${APRUN_default} --cpus-per-task=${NTHREADS_OCNANAL}"
+
+    export APRUN_MARINEANLOBSSTATS="${launcher} -n 1"
  ;;
  "marineanlletkf")
 
diff --git a/env/ORION.env b/env/ORION.env
index 3031a5e16a5..a3408529ade 100755
--- a/env/ORION.env
+++ b/env/ORION.env
@@ -148,6 +148,8 @@ elif [[ "${step}" = "marineanlchkpt" ]]; then
 
     export APRUN_MARINEANLCHKPT="${APRUN_default} --cpus-per-task=${NTHREADS_OCNANAL}"
 
+    export APRUN_MARINEANLOBSSTATS="${launcher} -n 1"
+
 elif [[ "${step}" = "ocnanalecen" ]]; then
 
     export NTHREADS_OCNANALECEN=${NTHREADSmax}
diff --git a/env/WCOSS2.env b/env/WCOSS2.env
index 16e04057e45..7333a127dd4 100755
--- a/env/WCOSS2.env
+++ b/env/WCOSS2.env
@@ -124,6 +124,8 @@ elif [[ "${step}" = "marineanlchkpt" ]]; then
 
     export APRUN_MARINEANLCHKPT="${APRUN_default}"
 
+    export APRUN_MARINEANLOBSSTATS="${launcher} -n 1"
+
 elif [[ "${step}" = "ocnanalecen" ]]; then
 
     export NTHREADS_OCNANALECEN=${NTHREADSmax}
diff --git a/parm/archive/gdasocean_analysis.yaml.j2 b/parm/archive/gdasocean_analysis.yaml.j2
index 7652d7bebdb..38fc09c3d3d 100644
--- a/parm/archive/gdasocean_analysis.yaml.j2
+++ b/parm/archive/gdasocean_analysis.yaml.j2
@@ -7,7 +7,7 @@ gdasocean_analysis:
         - '{{ COMIN_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}ocninc.nc'
         {% for domain in ["ocn", "ice"] %}
         - '{{ COMIN_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}{{domain}}.incr.nc'
-        - '{{ COMIN_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}{{domain}}ana.nc'
+        - '{{ COMIN_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}{{domain}}.ana.nc'
         {% endfor %}
 
         # static background error
@@ -26,7 +26,7 @@ gdasocean_analysis:
         {% endif %}
 
         # obs space diags
-        - '{{ COMIN_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}ocn.*.stats.csv'
+        - '{{ COMIN_OCEAN_ANALYSIS | relpath(ROTDIR) }}/diags/{{ head }}ocn.*.stats.csv'
         - '{{ COMIN_OCEAN_ANALYSIS | relpath(ROTDIR) }}/diags/*.nc4'
 
         # runtime configs
diff --git a/parm/gdas/soca_analysis_jedi_config.yaml.j2 b/parm/gdas/soca_analysis_jedi_config.yaml.j2
new file mode 100644
index 00000000000..123b3be5f8c
--- /dev/null
+++ b/parm/gdas/soca_analysis_jedi_config.yaml.j2
@@ -0,0 +1,26 @@
+var:
+  rundir: '{{ DATA }}'
+  exe_src: '{{ EXECgfs }}/gdas.x'
+  mpi_cmd: '{{ APRUN_MARINEANLVAR }}'
+  jedi_args: ['soca', 'variational']
+  jcb_base_yaml: '{{ PARMgfs }}/gdas/soca/jcb-base.yaml.j2'
+  jcb_algo_yaml: '{{ JCB_ALGO_YAML_VAR }}'
+socaincr2mom6:
+  rundir: '{{ DATA }}'
+  exe_src: '{{ EXECgfs }}/gdas_incr_handler.x'
+  mpi_cmd: '{{ APRUN_MARINEANLCHKPT }}'
+  jcb_base_yaml: '{{ PARMgfs }}/gdas/soca/jcb-base.yaml.j2'
+  jcb_algo: socaincr2mom6
+soca_2cice_global:
+  rundir: '{{ DATA }}'
+  exe_src: '{{ EXECgfs }}/gdas.x'
+  mpi_cmd: '{{ APRUN_MARINEANLCHKPT }}'
+  jedi_args: ['soca', 'convertstate']
+  jcb_base_yaml: '{{ PARMgfs }}/gdas/soca/jcb-base.yaml.j2'
+  jcb_algo: soca_2cice_global
+soca_diag_stats:
+  rundir: '{{ DATA }}'
+  exe_src: '{{ EXECgfs }}/gdassoca_obsstats.x'
+  mpi_cmd: '{{ APRUN_MARINEANLOBSSTATS }}'
+  jcb_base_yaml: '{{ PARMgfs }}/gdas/soca/jcb-base.yaml.j2'
+  jcb_algo: soca_diag_stats
diff --git a/parm/gdas/soca_bmat_jedi_config.yaml.j2 b/parm/gdas/soca_bmat_jedi_config.yaml.j2
index 9517626b98e..2996e1069e1 100644
--- a/parm/gdas/soca_bmat_jedi_config.yaml.j2
+++ b/parm/gdas/soca_bmat_jedi_config.yaml.j2
@@ -2,48 +2,48 @@ gridgen:
   rundir: '{{ DATA }}'
   exe_src: '{{ EXECgfs }}/gdas_soca_gridgen.x'
   mpi_cmd: '{{ APRUN_MARINEBMAT }}'
-  jcb_base_yaml: '{{ PARMgfs }}/gdas/soca/marine-jcb-base.yaml'
+  jcb_base_yaml: '{{ PARMgfs }}/gdas/soca/jcb-base.yaml.j2'
   jcb_algo: soca_gridgen
 soca_diagb:
   rundir: '{{ DATA }}'
   exe_src: '{{ EXECgfs }}/gdas_soca_diagb.x'
   mpi_cmd: '{{ APRUN_MARINEBMAT }}'
-  jcb_base_yaml: '{{ PARMgfs }}/gdas/soca/marine-jcb-base.yaml'
+  jcb_base_yaml: '{{ PARMgfs }}/gdas/soca/jcb-base.yaml.j2'
   jcb_algo: soca_diagb
 soca_chgres:
   rundir: '{{ DATA }}'
   exe_src: '{{ EXECgfs }}/gdas.x'
   mpi_cmd: '{{ APRUN_MARINEBMAT }}'
   jedi_args: ['soca', 'convertstate']
-  jcb_base_yaml: '{{ PARMgfs }}/gdas/soca/marine-jcb-base.yaml'
+  jcb_base_yaml: '{{ PARMgfs }}/gdas/soca/jcb-base.yaml.j2'
   jcb_algo: soca_chgres
 soca_parameters_diffusion_vt:
   rundir: '{{ DATA }}'
   exe_src: '{{ EXECgfs }}/gdas_soca_error_covariance_toolbox.x'
   mpi_cmd: '{{ APRUN_MARINEBMAT }}'
-  jcb_base_yaml: '{{ PARMgfs }}/gdas/soca/marine-jcb-base.yaml'
+  jcb_base_yaml: '{{ PARMgfs }}/gdas/soca/jcb-base.yaml.j2'
   jcb_algo: soca_parameters_diffusion_vt
 soca_setcorscales:
   rundir: '{{ DATA }}'
   exe_src: '{{ EXECgfs }}/gdas_soca_setcorscales.x'
   mpi_cmd: '{{ APRUN_MARINEBMAT }}'
-  jcb_base_yaml: '{{ PARMgfs }}/gdas/soca/marine-jcb-base.yaml'
+  jcb_base_yaml: '{{ PARMgfs }}/gdas/soca/jcb-base.yaml.j2'
   jcb_algo: soca_setcorscales
 soca_parameters_diffusion_hz:
   rundir: '{{ DATA }}'
   exe_src: '{{ EXECgfs }}/gdas_soca_error_covariance_toolbox.x'
   mpi_cmd: '{{ APRUN_MARINEBMAT }}'
-  jcb_base_yaml: '{{ PARMgfs }}/gdas/soca/marine-jcb-base.yaml'
+  jcb_base_yaml: '{{ PARMgfs }}/gdas/soca/jcb-base.yaml.j2'
   jcb_algo: soca_parameters_diffusion_hz
 soca_ensb:
   rundir: '{{ DATA }}'
   exe_src: '{{ EXECgfs }}/gdas_ens_handler.x'
   mpi_cmd: '{{ APRUN_MARINEBMAT }}'
-  jcb_base_yaml: '{{ PARMgfs }}/gdas/soca/marine-jcb-base.yaml'
+  jcb_base_yaml: '{{ PARMgfs }}/gdas/soca/jcb-base.yaml.j2'
   jcb_algo: soca_ensb
 soca_ensweights:
   rundir: '{{ DATA }}'
   exe_src: '{{ EXECgfs }}/gdas_socahybridweights.x'
   mpi_cmd: '{{ APRUN_MARINEBMAT }}'
-  jcb_base_yaml: '{{ PARMgfs }}/gdas/soca/marine-jcb-base.yaml'
+  jcb_base_yaml: '{{ PARMgfs }}/gdas/soca/jcb-base.yaml.j2'
   jcb_algo: soca_ensweights
diff --git a/parm/gdas/stat_base_config.yaml.j2 b/parm/gdas/stat_base_config.yaml.j2
index 06badfd9ca2..37f5f6641aa 100644
--- a/parm/gdas/stat_base_config.yaml.j2
+++ b/parm/gdas/stat_base_config.yaml.j2
@@ -24,3 +24,9 @@ snow:
   - name: ims_snow
     input file: "diag_ims_snow_{{ current_cycle | to_YMDH }}.nc"
     output file: "ims_snow_{{ current_cycle | to_YMDH }}_output_snow.nc"
+  - name: sfcsno
+    input file: "diag_sfcsno_{{ current_cycle | to_YMDH }}.nc"
+    output file: "sfcsno_{{ current_cycle | to_YMDH }}_output_snow.nc"
+  - name: snocvr
+    input file: "diag_snocvr_{{ current_cycle | to_YMDH }}.nc"
+    output file: "snocvr_{{ current_cycle | to_YMDH }}_output_snow.nc"
diff --git a/scripts/exglobal_marine_analysis_checkpoint.py b/scripts/exglobal_marine_analysis_checkpoint.py
index c47a2e3a0e9..1690bcff57f 100755
--- a/scripts/exglobal_marine_analysis_checkpoint.py
+++ b/scripts/exglobal_marine_analysis_checkpoint.py
@@ -22,7 +22,10 @@
     MarineAnl = MarineAnalysis(config)
 
     # Prepare the SOCA increment for MOM6 IAU
-    MarineAnl.checkpoint_mom6_iau('socaincr2mom6.yaml')
+    MarineAnl.execute('socaincr2mom6')
 
     # Insert the seaice analysis into the CICE6 restart
-    MarineAnl.checkpoint_cice6('soca_2cice_global.yaml')
+    MarineAnl.execute('soca_2cice_global')
+
+    # Compute the observation space statistics
+    MarineAnl.execute('soca_diag_stats')
diff --git a/scripts/exglobal_marine_analysis_finalize.py b/scripts/exglobal_marine_analysis_finalize.py
index daa3fbb487d..d65dfe8bdf8 100755
--- a/scripts/exglobal_marine_analysis_finalize.py
+++ b/scripts/exglobal_marine_analysis_finalize.py
@@ -22,6 +22,3 @@
 
     # Make a copy of the analysis output to the COMROOT
     MarineAnl.finalize()
-
-    # Compute the observation space statistics
-    MarineAnl.obs_space_stats()
diff --git a/scripts/exglobal_marine_analysis_variational.py b/scripts/exglobal_marine_analysis_variational.py
index e03c56b1e51..8d5cf2347f9 100755
--- a/scripts/exglobal_marine_analysis_variational.py
+++ b/scripts/exglobal_marine_analysis_variational.py
@@ -21,4 +21,4 @@
     MarineAnl = MarineAnalysis(config)
 
     # Run the variational application
-    MarineAnl.variational()
+    MarineAnl.execute('var')
diff --git a/sorc/gdas.cd b/sorc/gdas.cd
index 07d8d2c2747..cb4f83971cc 160000
--- a/sorc/gdas.cd
+++ b/sorc/gdas.cd
@@ -1 +1 @@
-Subproject commit 07d8d2c27477ec436bd900613615c319df038c21
+Subproject commit cb4f83971cc8d2e3914c8b1db9ebff886fcd27fd
diff --git a/ush/python/pygfs/jedi/jedi.py b/ush/python/pygfs/jedi/jedi.py
index 2806ba4bce9..06986ec7d37 100644
--- a/ush/python/pygfs/jedi/jedi.py
+++ b/ush/python/pygfs/jedi/jedi.py
@@ -70,7 +70,7 @@ def __init__(self, config: Dict[str, Any]) -> None:
         self._jedi_config = self.jedi_config.deepcopy()
 
     @logit(logger)
-    def initialize(self, task_config: AttrDict) -> None:
+    def initialize(self, task_config: AttrDict, clean_empty_obsspaces=False) -> None:
         """Initialize JEDI application
 
         This method will initialize a JEDI application.
@@ -83,6 +83,9 @@ def initialize(self, task_config: AttrDict) -> None:
         ----------
         task_config: AttrDict
             Attribute-dictionary of all configuration variables associated with a GDAS task.
+        clean_empty_obsspaces: bool
+            Flag to clean empty observation spaces from JEDI input configuration dictionary.
+            Default is False.
 
         Returns
         ----------
@@ -94,6 +97,11 @@ def initialize(self, task_config: AttrDict) -> None:
         self.jedi_config.input_config = self.render_jcb(task_config)
         logger.debug(f"JEDI config:\n{pformat(self.jedi_config.input_config)}")
 
+        # Remove obs spaces from JEDI config dictionary with missing obs files
+        if clean_empty_obsspaces:
+            logger.info(f"Clean empty obs spaces from JEDI YAML config: {self.jedi_config.yaml}")
+            self.clean_empty_obsspaces()
+
         # Save JEDI config dictionary to YAML in run directory
         logger.debug(f"Writing JEDI YAML config to: {self.jedi_config.yaml}")
         save_as_yaml(self.jedi_config.input_config, self.jedi_config.yaml)
@@ -257,6 +265,44 @@ def get_jedi_dict(jedi_config_yaml: str, task_config: AttrDict, expected_block_n
         # Return dictionary of JEDI objects
         return jedi_dict
 
+    @logit(logger)
+    def clean_empty_obsspaces(self):
+        """
+        Replace list of observers in JEDI input configuration dictionary with new list, removing
+        any observers with missing observation files.
+
+        Parameters
+        ----------
+        None
+
+        Returns
+        -------
+        None
+        """
+
+        # Get observers from JEDI input config
+        observers = find_value_in_nested_dict(self.jedi_config.input_config, 'observers')
+
+        # Check if observers list actually present
+        if observers:
+            # Create new list of observers
+            cleaned_observers = []
+            for obs_space in observers:
+                fname = obs_space['obs space']['obsdatain']['engine']['obsfile']
+                if os.path.isfile(fname):
+                    cleaned_observers.append(obs_space)
+                else:
+                    logger.warning(f"WARNING: {fname} does not exist, removing obs space")
+
+            # Clear observers list in dictionary and replace with new list
+            observers.clear()
+            observers.extend(cleaned_observers)
+
+            # If no observers left in list, raise error
+            if observers == []:
+                logger.error(f"FATAL ERROR: No observers found in JEDI input config")
+                raise Exception(f"FATAL ERROR: No observers found in JEDI input config")
+
     @staticmethod
     @logit(logger)
     def remove_redundant(input_list: List) -> List:
@@ -349,4 +395,76 @@ def extract_tar(tar_file: str) -> None:
             raise tarfile.ReadError(f"FATAL ERROR: {tar_file} is not a tar archive")
     except tarfile.ExtractError as err:
         logger.exception(f"FATAL ERROR: unable to extract from {tar_file}")
-        raise tarfile.ExtractError("FATAL ERROR: unable to extract from {tar_file}")
+        raise tarfile.ExtractError(f"FATAL ERROR: unable to extract from {tar_file}")
+
+
+@logit(logger)
+def find_value_in_nested_dict(nested_dict: Dict, target_key: str) -> Any:
+    """
+    Recursively search through a nested dictionary and return the value for the target key.
+    This returns the first target key it finds.  So if a key exists in a subsequent
+    nested dictionary, it will not be found.
+
+    Parameters
+    ----------
+    nested_dict : Dict
+        Dictionary to search
+    target_key : str
+        Key to search for
+
+    Returns
+    -------
+    Any
+        Value of the target key
+
+    Raises
+    ------
+    KeyError
+        If key is not found in dictionary
+
+    TODO: if this gives issues due to landing on an incorrect key in the nested
+    dictionary, we will have to implement a more concrete method to search for a key
+    given a more complete address.  See resolved conversations in PR 2387
+
+    # Example usage:
+    nested_dict = {
+        'a': {
+            'b': {
+                'c': 1,
+                'd': {
+                    'e': 2,
+                    'f': 3
+                }
+            },
+            'g': 4
+        },
+        'h': {
+            'i': 5
+        },
+        'j': {
+            'k': 6
+        }
+    }
+
+    user_key = input("Enter the key to search for: ")
+    result = find_value_in_nested_dict(nested_dict, user_key)
+    """
+
+    if not isinstance(nested_dict, dict):
+        raise TypeError(f"Input is not of type(dict)")
+
+    result = nested_dict.get(target_key)
+    if result is not None:
+        return result
+
+    for value in nested_dict.values():
+        if isinstance(value, dict):
+            try:
+                result = find_value_in_nested_dict(value, target_key)
+                if result is not None:
+                    return result
+            except KeyError:
+                pass
+
+    logger.info(f"Key '{target_key}' not found in the nested dictionary")
+    return None
diff --git a/ush/python/pygfs/task/aero_analysis.py b/ush/python/pygfs/task/aero_analysis.py
index b88ee6e4802..7b47e441343 100644
--- a/ush/python/pygfs/task/aero_analysis.py
+++ b/ush/python/pygfs/task/aero_analysis.py
@@ -94,10 +94,6 @@ def initialize(self) -> None:
         - creating output directories
         """
 
-        # initialize JEDI variational application
-        logger.info(f"Initializing JEDI variational DA application")
-        self.jedi_dict['aeroanlvar'].initialize(self.task_config)
-
         # stage observations
         logger.info(f"Staging list of observation files generated from JEDI config")
         obs_dict = self.jedi_dict['aeroanlvar'].render_jcb(self.task_config, 'aero_obs_staging')
@@ -135,6 +131,10 @@ def initialize(self) -> None:
         FileHandler(aero_var_stage_dict).sync()
         logger.debug(f"Staging from COM:\n{pformat(aero_var_stage_dict)}")
 
+        # initialize JEDI variational application
+        logger.info(f"Initializing JEDI variational DA application")
+        self.jedi_dict['aeroanlvar'].initialize(self.task_config, clean_empty_obsspaces=True)
+
     @logit(logger)
     def execute(self, jedi_dict_key: str) -> None:
         """Execute JEDI application of aero analysis
diff --git a/ush/python/pygfs/task/atm_analysis.py b/ush/python/pygfs/task/atm_analysis.py
index 7879c6741e4..0259c016188 100644
--- a/ush/python/pygfs/task/atm_analysis.py
+++ b/ush/python/pygfs/task/atm_analysis.py
@@ -98,14 +98,6 @@ def initialize(self) -> None:
         None
         """
 
-        # initialize JEDI variational application
-        logger.info(f"Initializing JEDI variational DA application")
-        self.jedi_dict['atmanlvar'].initialize(self.task_config)
-
-        # initialize JEDI FV3 increment conversion application
-        logger.info(f"Initializing JEDI FV3 increment conversion application")
-        self.jedi_dict['atmanlfv3inc'].initialize(self.task_config)
-
         # stage observations
         logger.info(f"Staging list of observation files")
         obs_dict = self.jedi_dict['atmanlvar'].render_jcb(self.task_config, 'atm_obs_staging')
@@ -167,6 +159,14 @@ def initialize(self) -> None:
         ]
         FileHandler({'mkdir': newdirs}).sync()
 
+        # initialize JEDI variational application
+        logger.info(f"Initializing JEDI variational DA application")
+        self.jedi_dict['atmanlvar'].initialize(self.task_config, clean_empty_obsspaces=True)
+
+        # initialize JEDI FV3 increment conversion application
+        logger.info(f"Initializing JEDI FV3 increment conversion application")
+        self.jedi_dict['atmanlfv3inc'].initialize(self.task_config)
+
     @logit(logger)
     def execute(self, jedi_dict_key: str) -> None:
         """Execute JEDI application of atm analysis
diff --git a/ush/python/pygfs/task/atmens_analysis.py b/ush/python/pygfs/task/atmens_analysis.py
index 1640a815db8..b146b86360e 100644
--- a/ush/python/pygfs/task/atmens_analysis.py
+++ b/ush/python/pygfs/task/atmens_analysis.py
@@ -78,7 +78,7 @@ def initialize(self) -> None:
 
         This method will initialize a global atmens analysis.
         This includes:
-        - initialize JEDI applications
+        - initialize JEDI LETKF observer and FV3 increment converter applications
         - staging observation files
         - staging bias correction files
         - staging CRTM fix files
@@ -95,18 +95,6 @@ def initialize(self) -> None:
         None
         """
 
-        # initialize JEDI LETKF observer application
-        logger.info(f"Initializing JEDI LETKF observer application")
-        self.jedi_dict['atmensanlobs'].initialize(self.task_config)
-
-        # initialize JEDI LETKF solver application
-        logger.info(f"Initializing JEDI LETKF solver application")
-        self.jedi_dict['atmensanlsol'].initialize(self.task_config)
-
-        # initialize JEDI FV3 increment conversion application
-        logger.info(f"Initializing JEDI FV3 increment conversion application")
-        self.jedi_dict['atmensanlfv3inc'].initialize(self.task_config)
-
         # stage observations
         logger.info(f"Staging list of observation files")
         obs_dict = self.jedi_dict['atmensanlobs'].render_jcb(self.task_config, 'atm_obs_staging')
@@ -149,6 +137,18 @@ def initialize(self) -> None:
         ]
         FileHandler({'mkdir': newdirs}).sync()
 
+        # initialize JEDI LETKF observer application
+        logger.info(f"Initializing JEDI LETKF observer application")
+        self.jedi_dict['atmensanlobs'].initialize(self.task_config, clean_empty_obsspaces=True)
+
+        # initialize JEDI LETKF solver application
+        logger.info(f"Initializing JEDI LETKF solver application")
+        self.jedi_dict['atmensanlsol'].initialize(self.task_config)
+
+        # initialize JEDI FV3 increment conversion application
+        logger.info(f"Initializing JEDI FV3 increment conversion application")
+        self.jedi_dict['atmensanlfv3inc'].initialize(self.task_config)
+
     @logit(logger)
     def initialize_letkf(self) -> None:
         """Initialize a global atmens analysis
diff --git a/ush/python/pygfs/task/marine_analysis.py b/ush/python/pygfs/task/marine_analysis.py
index f45f1162641..3d297d8158e 100644
--- a/ush/python/pygfs/task/marine_analysis.py
+++ b/ush/python/pygfs/task/marine_analysis.py
@@ -1,26 +1,15 @@
 #!/usr/bin/env python3
 
-import copy
-import os
+from datetime import datetime
 from logging import getLogger
+import os
 import pygfs.utils.marine_da_utils as mdau
-import glob
-import re
-import netCDF4
-from multiprocessing import Process
-import subprocess
-import yaml
-from jcb import render
-
-from wxflow import (AttrDict,
-                    FileHandler,
-                    add_to_datetime, to_timedelta, to_YMD,
-                    parse_j2yaml, parse_yaml,
-                    logit,
-                    Executable,
-                    Task,
-                    save_as_yaml,
-                    Template, TemplateConstants, YAMLFile)
+from pygfs.jedi import Jedi
+
+from wxflow import (AttrDict, FileHandler, Task,
+                    add_to_datetime, to_isotime, to_timedelta, to_YMD,
+                    parse_j2yaml,
+                    logit)
 
 logger = getLogger(__name__.split('.')[-1])
 
@@ -31,6 +20,23 @@ class MarineAnalysis(Task):
     """
     @logit(logger, name="MarineAnalysis")
     def __init__(self, config):
+        """Constructor for global marine analysis
+
+        This method will construct a marine analysis task
+        This includes:
+        - extending the task_config attribute AttrDict to include parameters required for this task
+        - instantiate the dictionary of Jedi objects for each JEDI application
+
+        Parameters
+        ----------
+        config: Dict
+            dictionary object containing task configuration
+
+        Returns
+        ----------
+        None
+        """
+
         super().__init__(config)
         _calc_scale_exec = os.path.join(self.task_config.HOMEgfs, 'ush', 'soca', 'calc_scales.py')
         _window_begin = add_to_datetime(self.task_config.current_cycle, -to_timedelta(f"{self.task_config.assim_freq}H") / 2)
@@ -42,130 +48,64 @@ def __init__(self, config):
         else:
             _enspert_relpath = None
 
+        # Determine background error model
+        if self.task_config.NMEM_ENS >= 2:
+            _berror_model = 'marine_background_error_hybrid_diffusion_diffusion'
+        else:
+            _berror_model = 'marine_background_error_static_diffusion'
+
         # Create a local dictionary that is repeatedly used across this class
         local_dict = AttrDict(
             {
                 'PARMsoca': os.path.join(self.task_config.PARMgfs, 'gdas', 'soca'),
                 'MARINE_WINDOW_BEGIN': _window_begin,
-                'MARINE_WINDOW_BEGIN_ISO': _window_begin.strftime('%Y-%m-%dT%H:%M:%SZ'),
                 'MARINE_WINDOW_END': _window_end,
-                'MARINE_WINDOW_LENGTH': f"PT{self.task_config['assim_freq']}H",
                 'MARINE_WINDOW_MIDDLE': self.task_config.current_cycle,
-                'MARINE_WINDOW_MIDDLE_ISO': self.task_config.current_cycle.strftime('%Y-%m-%dT%H:%M:%SZ'),
+                'MARINE_WINDOW_LENGTH': f"PT{self.task_config['assim_freq']}H",
+                'MARINE_WINDOW_BEGIN_ISO': to_isotime(_window_begin),
+                'MARINE_WINDOW_MIDDLE_ISO': to_isotime(self.task_config.current_cycle),
                 'ENSPERT_RELPATH': _enspert_relpath,
                 'CALC_SCALE_EXEC': _calc_scale_exec,
                 'OPREFIX': f"{self.task_config.RUN}.t{self.task_config.cyc:02d}z.",
                 'APREFIX': f"{self.task_config.RUN}.t{self.task_config.cyc:02d}z.",
-                'app_path_observations': self.task_config.MARINE_JCB_GDAS_OBS
+                'berror_model': _berror_model,
+                'MOM6_LEVS': mdau.get_mom6_levels(str(self.task_config.OCNRES).zfill(3)),
+                'app_path_observations': self.task_config.MARINE_JCB_GDAS_OBS,
+                'marine_pseudo_model_states': mdau.gen_bkg_list(bkg_path='./bkg',
+                                                                window_begin=_window_begin)
             }
         )
 
         # Extend task_config with local_dict
         self.task_config.update(local_dict)
 
+        # Construct dictionary of JEDI objects, one for each JEDI application need for the analysis
+        expected_keys = ['var', 'socaincr2mom6', 'soca_2cice_global', 'soca_diag_stats']
+        self.jedi_dict = Jedi.get_jedi_dict(self.task_config.JEDI_CONFIG_YAML_ANALYSIS, self.task_config, expected_keys)
+
     @logit(logger)
     def initialize(self: Task) -> None:
-        """Initialize the marine analysis
+        """Initialize the marine analysis task
 
         This method will initialize the marine analysis.
         This includes:
-        - staging the deterministic backgrounds (middle of window)
         - staging SOCA fix files
-        - staging static ensemble members (optional)
-        - staging ensemble members (optional)
-        - generating the YAML files for the JEDI and GDASApp executables
-        - creating output directories
-        """
-        super().initialize()
-
-        # prepare the directory structure to run SOCA
-        self._prep_scratch_dir()
-
-        # fetch observations from COMROOT
-        # TODO(G.V. or A.E.): Keep a copy of the obs in the scratch fs after the obs prep job
-        self._fetch_observations()
-
-        # stage the ocean and ice backgrounds for FGAT
-        bkg_list = parse_j2yaml(self.task_config.MARINE_DET_STAGE_BKG_YAML_TMPL, self.task_config)
-        FileHandler(bkg_list).sync()
-
-        # stage the soca grid
-        FileHandler({'copy': [[os.path.join(self.task_config.COMIN_OCEAN_BMATRIX, 'soca_gridspec.nc'),
-                               os.path.join(self.task_config.DATA, 'soca_gridspec.nc')]]}).sync()
-
-        # link the flow dependent static B resources from the B-matrix task of the same cycle
-        os.symlink('../staticb', 'staticb')
-
-        # hybrid EnVAR case
-        if self.task_config.DOHYBVAR_OCN == "YES" or self.task_config.NMEM_ENS >= 2:
-            # stage the ensemble weights
-            logger.debug(f"Stage ensemble weights for the hybrid background error")
-            FileHandler({'copy': [[os.path.join(self.task_config.COMIN_OCEAN_BMATRIX, f'{self.task_config.APREFIX}ocean.ens_weights.nc'),
-                                   os.path.join(self.task_config.DATA, 'ocean.ens_weights.nc')],
-                                  [os.path.join(self.task_config.COMIN_ICE_BMATRIX, f'{self.task_config.APREFIX}ice.ens_weights.nc'),
-                                   os.path.join(self.task_config.DATA, 'ice.ens_weights.nc')]]}).sync()
-
-        # prepare the yaml configuration to run the SOCA variational application
-        self._prep_variational_yaml()
-
-        # prepare the yaml configuration to run the SOCA to MOM6 IAU increment
-        self._prep_checkpoint()
-
-    @logit(logger)
-    def _fetch_observations(self: Task) -> None:
-        """Fetch observations from COMIN_OBS
-
-        This method will fetch the observations for the cycle and check the
-        list against what is available for the cycle.
-        """
-
-        # get the list of observations
-
-        # "observations" is expected by later JCB code to populate it with config info,
-        # but the obs_list as such is needed later
-        self.task_config.observations = parse_j2yaml(self.task_config.MARINE_OBS_LIST_YAML, self.task_config)['observations']
-        self.task_config.obs_list = self.task_config.observations
-
-        obsconfigfile = os.path.join(self.task_config['PARMgfs'], 'gdas/soca/obs/obs_list_base_yaml.j2')
-        self.task_config.observations = parse_j2yaml(obsconfigfile, self.task_config)
-
-        obs_files = []
-
-        for observer in self.task_config['observations']['observers']:
-            filename = f"{self.task_config.OPREFIX}{observer['obs space']['name'].lower()}.{to_YMD(self.task_config.PDY)}{self.task_config.cyc:02d}.nc4"
-            logger.info(f"******** {filename}")
-            obs_files.append(filename)
-
-        obs_files_to_copy = []
-
-        # copy obs from COMIN_OBS to DATA/obs
-        for obs_file in obs_files:
-            logger.info(f"******* {obs_file}")
-            obs_src = os.path.join(self.task_config.COMIN_OBS, obs_file)
-            obs_dst = os.path.join(self.task_config.DATA, 'obs', obs_file)
-            logger.info(f"******* {obs_src}")
-            if os.path.exists(obs_src):
-                logger.info(f"******* fetching {obs_file}")
-                obs_files_to_copy.append([obs_src, obs_dst])
-            else:
-                logger.info(f"******* {obs_file} is not in the database")
+        - preparing the namelists for deterministic MOM6 and analysis geometry
+        - staging observations
+        - staging input YAMLs for SOCA utilities
+        - staging the deterministic backgrounds (middle of window)
+        - staging files and link directories from B-matrix job needed for deterministic analysis
+        - generating list of model pseudo states
+        - initializing all the JEDI applications required for the marine analysis
 
-        FileHandler({'copy': obs_files_to_copy}).sync()
+        Parameters
+        ----------
+        None
 
-    @logit(logger)
-    def _prep_scratch_dir(self: Task) -> None:
-        """Create and stage all the resources needed to run SOCA/JEDI, including the necesssary
-           directory structure to run the SOCA variational application
+        Returns
+        ----------
+        None
         """
-        logger.info(f"---------------- Setup runtime environement")
-
-        anl_dir = self.task_config.DATA
-
-        # create analysis directories
-        diags = os.path.join(anl_dir, 'diags')            # output dir for soca DA obs space
-        obs_in = os.path.join(anl_dir, 'obs')             # input      "           "
-        anl_out = os.path.join(anl_dir, 'Data')           # output dir for soca DA
-        FileHandler({'mkdir': [diags, obs_in, anl_out]}).sync()
 
         # stage fix files
         logger.info(f"Staging SOCA fix files from {self.task_config.SOCA_INPUT_FIX_DIR}")
@@ -173,321 +113,129 @@ def _prep_scratch_dir(self: Task) -> None:
         FileHandler(soca_fix_list).sync()
 
         # prepare the deterministic MOM6 input.nml
+        logger.info(f"Preparing deterministic MOM6 input namelist")
         mdau.prep_input_nml(self.task_config)
 
         # prepare the input.nml for the analysis geometry
+        logger.info(f"Preparing analysis geometry input namelist")
         mdau.prep_input_nml(self.task_config, output_nml="./anl_geom/mom_input.nml",
                             simple_geom=True, mom_input="./anl_geom/MOM_input")
 
+        # fetch observations from COMROOT
+        # TODO(G.V. or A.E.): Keep a copy of the obs in the scratch fs after the obs prep job
+        logger.info(f"Staging observations from {self.task_config.COMIN_OBS}")
+        obs_list = self.jedi_dict['var'].render_jcb(self.task_config, 'soca_obs_staging')
+        FileHandler(obs_list).sync()
+
         # stage the soca utility yamls (gridgen, fields and ufo mapping yamls)
         logger.info(f"Staging SOCA utility yaml files from {self.task_config.PARMsoca}")
         soca_utility_list = parse_j2yaml(self.task_config.MARINE_UTILITY_YAML_TMPL, self.task_config)
         FileHandler(soca_utility_list).sync()
 
-    @logit(logger)
-    def _prep_variational_yaml(self: Task) -> None:
-        """Create the yaml configuration to run the SOCA variational application
-        """
+        # stage the ocean and ice backgrounds for FGAT
+        logger.info(f"Staging files needed for deterministic analysis from COM")
+        bkg_list = parse_j2yaml(self.task_config.MARINE_DET_STAGE_BKG_YAML_TMPL, self.task_config)
+        FileHandler(bkg_list).sync()
 
-        # prepare background list for the pseudo model, check bkg date for consistency
-        mdau.gen_bkg_list(bkg_path='./bkg',
-                          window_begin=self.task_config.MARINE_WINDOW_BEGIN,
-                          yaml_name='bkg_list.yaml')
+        # stage files and link directories from B-matrix job needed for deterministic analysis
+        logger.info(f"Staging files needed for deterministic analysis from COM")
+        soca_files_list = parse_j2yaml(self.task_config.MARINE_DET_STAGE_FILES_YAML_TMPL, self.task_config)
+        FileHandler(soca_files_list).sync()
 
-        # Make a copy of the env config before modifying to avoid breaking something else
-        envconfig_jcb = copy.deepcopy(self.task_config)
-        logger.info(f"---------------- Prepare the yaml configuration")
-        logger.info(f"{envconfig_jcb}")       # Prepare the yaml configuration
+        # assert that dates of the history files are correct
+        mdau.test_hist_date('./INPUT/MOM.res.nc', self.task_config.MARINE_WINDOW_BEGIN)
+        for state in self.task_config.marine_pseudo_model_states:
+            mdau.test_hist_date(state['basename'] + state['ocn_filename'],
+                                datetime.strptime(state['date'], '%Y-%m-%dT%H:%M:%SZ'))
 
-        # Add the things to the envconfig in order to template JCB files
-        envconfig_jcb['PARMgfs'] = self.task_config.PARMgfs
-        envconfig_jcb['NMEM_ENS'] = self.task_config.NMEM_ENS
-        envconfig_jcb['berror_model'] = 'marine_background_error_static_diffusion'
-        if self.task_config.NMEM_ENS >= 2:
-            envconfig_jcb['berror_model'] = 'marine_background_error_hybrid_diffusion_diffusion'
-        envconfig_jcb['DATA'] = self.task_config.DATA
-        envconfig_jcb['OPREFIX'] = self.task_config.OPREFIX
-        envconfig_jcb['PDY'] = os.getenv('PDY')
-        envconfig_jcb['cyc'] = os.getenv('cyc')
-        envconfig_jcb['SOCA_NINNER'] = self.task_config.SOCA_NINNER
-        envconfig_jcb['HOMEgfs'] = self.task_config.HOMEgfs
-        envconfig_jcb['DO_TEST_MODE'] = self.task_config.DO_TEST_MODE
-        envconfig_jcb['RUN'] = self.task_config.RUN
-        envconfig_jcb['current_cycle'] = self.task_config.current_cycle
-        envconfig_jcb['MOM6_LEVS'] = mdau.get_mom6_levels(str(self.task_config.OCNRES).zfill(3))
-        envconfig_jcb['observations'] = self.task_config.observations
-
-        # Write obs_list_short
-        save_as_yaml(self.task_config['obs_list'], 'obs_list_short.yaml')
-        os.environ['OBS_LIST_SHORT'] = 'obs_list_short.yaml'
-
-        # Render the JCB configuration files
-        jcb_base_yaml = os.path.join(self.task_config.PARMsoca, 'marine-jcb-base.yaml')
-        jcb_algo_yaml = self.task_config.JCB_ALGO_YAML_VAR
-
-        jcb_base_config = parse_j2yaml(path=jcb_base_yaml, data=envconfig_jcb)
-        jcb_algo_config = parse_j2yaml(path=jcb_algo_yaml, data=envconfig_jcb)
-
-        # Override base with the application specific config
-        jcb_config = {**jcb_base_config, **jcb_algo_config}
-
-        # convert datetime to string
-        jcb_config['window_begin'] = self.task_config.MARINE_WINDOW_BEGIN.strftime('%Y-%m-%dT%H:%M:%SZ')
-        jcb_config['window_middle'] = self.task_config.MARINE_WINDOW_MIDDLE.strftime('%Y-%m-%dT%H:%M:%SZ')
-
-        # Current hack so that this is not done directly in the JCB base yaml
-        jcb_config['marine_pseudo_model_states'] = parse_yaml('bkg_list.yaml')
-
-        # Render the full JEDI configuration file using JCB
-        jedi_config = render(jcb_config)
-
-        # Save the JEDI configuration file
-        var_yaml_jcb = 'var.yaml'
-        mdau.clean_empty_obsspaces(jedi_config, target=var_yaml_jcb, app='var')
-
-    def _prep_checkpoint(self: Task) -> None:
-        """Create the yaml configuration to run the SOCA to MOM6 IAU increment
-        """
-        # prepare the socaincr2mom6.yaml
-        logger.info("Generate the SOCA to MOM6 IAU increment YAML file")
-        data = {'marine_window_begin': self.task_config.MARINE_WINDOW_BEGIN_ISO,
-                'marine_window_middle': self.task_config.MARINE_WINDOW_MIDDLE_ISO}
-        soca2mom6inc_config = parse_j2yaml(path=os.path.join(self.task_config.MARINE_JCB_GDAS_ALGO, 'socaincr2mom6.yaml.j2'),
-                                           data=data)
-        soca2mom6inc_config.save(os.path.join(self.task_config.DATA, 'socaincr2mom6.yaml'))
-
-        # prepare the SOCA to CICE YAML file
-        logger.info("Generate the SOCA to CICE RST YAML file")
-
-        # set the restart date, dependent on the cycling type
-        if self.task_config.DOIAU:
-            # forecast initialized at the begining of the DA window
-            fcst_begin = self.task_config.MARINE_WINDOW_BEGIN_ISO
-            rst_date = self.task_config.MARINE_WINDOW_BEGIN.strftime('%Y%m%d.%H%M%S')
-        else:
-            # forecast initialized at the middle of the DA window
-            fcst_begin = self.task_config.MARINE_WINDOW_MIDDLE_ISO
-            rst_date = self.task_config.MARINE_WINDOW_MIDDLE.strftime('%Y%m%d.%H%M%S')
-
-        # make a copy of the CICE6 restart
-        ice_rst = os.path.join(self.task_config.COMIN_ICE_RESTART_PREV, f'{rst_date}.cice_model.res.nc')
-        ice_rst_ana = os.path.join(self.task_config.DATA, 'Data', rst_date + '.cice_model.res.nc')
-        FileHandler({'copy': [[ice_rst, ice_rst_ana]]}).sync()
-
-        # prepare the necessary configuration for the SOCA to CICE application
-        soca2cice_param = AttrDict({
-            "ocn_ana": f"./Data/ocn.3dvarfgat_pseudo.an.{self.task_config.MARINE_WINDOW_MIDDLE_ISO}.nc",
-            "ice_ana": f"./Data/ice.3dvarfgat_pseudo.an.{self.task_config.MARINE_WINDOW_MIDDLE_ISO}.nc",
-            "ocn_inc": f"./Data/ocn.3dvarfgat_pseudo.incr.{self.task_config.MARINE_WINDOW_MIDDLE_ISO}.nc",
-            "ice_inc": f"./Data/ice.3dvarfgat_pseudo.incr.{self.task_config.MARINE_WINDOW_MIDDLE_ISO}.nc",
-            "ice_rst": ice_rst_ana,
-            "fcst_begin": fcst_begin
-        })
-        logger.debug(f"{soca2cice_param}")
-
-        # render the SOCA to CICE YAML file for the Arctic and Antarctic
-        logger.info("render the SOCA to CICE YAML file for the Arctic and Antarctic")
-        varchgyaml = 'soca_2cice_global.yaml'
-        soca2cice_config = parse_j2yaml(path=os.path.join(self.task_config.MARINE_JCB_GDAS_ALGO, f'{varchgyaml}.j2'),
-                                        data=soca2cice_param)
-        soca2cice_config.save(os.path.join(self.task_config.DATA, varchgyaml))
+        # initialize JEDI applications
+        logger.info(f"Initializing SOCA variational application")
+        self.jedi_dict['var'].initialize(self.task_config, clean_empty_obsspaces=True)
 
-    @logit(logger)
-    def variational(self: Task) -> None:
-        # link gdas_soca_gridgen.x
-        mdau.link_executable(self.task_config, 'gdas.x')
-        exec_cmd = Executable(self.task_config.APRUN_MARINEANLVAR)
-        exec_name = os.path.join(self.task_config.DATA, 'gdas.x')
-        exec_cmd.add_default_arg(exec_name)
-        exec_cmd.add_default_arg('soca')
-        exec_cmd.add_default_arg('variational')
-        exec_cmd.add_default_arg('var.yaml')
-
-        mdau.run(exec_cmd)
+        logger.info(f"Initializing SOCA increment handler")
+        self.jedi_dict['socaincr2mom6'].initialize(self.task_config)
 
-    @logit(logger)
-    def checkpoint_cice6(self: Task, soca2ciceyaml) -> None:
-        # link gdas_soca_gridgen.x
-        mdau.link_executable(self.task_config, 'gdas.x')
-        exec_cmd = Executable(self.task_config.APRUN_MARINEANLCHKPT)
-        exec_name = os.path.join(self.task_config.DATA, 'gdas.x')
-        exec_cmd.add_default_arg(exec_name)
-        exec_cmd.add_default_arg('soca')
-        exec_cmd.add_default_arg('convertstate')
-        exec_cmd.add_default_arg(soca2ciceyaml)
-
-        mdau.run(exec_cmd)
+        logger.info(f"Initializing SOCA convertstate application")
+        self.jedi_dict['soca_2cice_global'].initialize(self.task_config)
+
+        # This method is a bit of a hack that will be removed in the future when the anlstat
+        # job fully replaces the SOCA obs_diag_stats application
+        self.initialize_obs_stats()
 
     @logit(logger)
-    def checkpoint_mom6_iau(self: Task, socaincr2mom6yaml) -> None:
-        # link gdas_incr_handler.x
-        mdau.link_executable(self.task_config, 'gdas_incr_handler.x')
-        exec_cmd = Executable(self.task_config.APRUN_MARINEANLCHKPT)
-        exec_name = os.path.join(self.task_config.DATA, 'gdas_incr_handler.x')
-        exec_cmd.add_default_arg(exec_name)
-        exec_cmd.add_default_arg(socaincr2mom6yaml)
+    def execute(self, jedi_dict_key: str) -> None:
+        """Execute JEDI application of marine analysis
 
-        mdau.run(exec_cmd)
+        Parameters
+        ----------
+        jedi_dict_key
+            key specifying particular Jedi object in self.jedi_dict
 
-    @logit(logger)
-    def finalize(self: Task) -> None:
-        """Finalize the marine analysis job
-           This method saves the results of the deterministic variational analysis to the COMROOT
+        Returns
+        ----------
+        None
         """
 
-        def list_all_files(dir_in, dir_out, wc='*', fh_list=[]):
-            files = glob.glob(os.path.join(dir_in, wc))
-            for file_src in files:
-                file_dst = os.path.join(dir_out, os.path.basename(file_src))
-                fh_list.append([file_src, file_dst])
-            return fh_list
-
-        # variables of convenience
-        com_ocean_analysis = self.task_config.COMOUT_OCEAN_ANALYSIS
-        com_ice_analysis = self.task_config.COMOUT_ICE_ANALYSIS
-        com_ice_restart = self.task_config.COMOUT_ICE_RESTART
-        anl_dir = self.task_config.DATA
-        cdate = self.task_config.CDATE
-        pdy = self.task_config.PDY
-        staticsoca_dir = self.task_config.SOCA_INPUT_FIX_DIR
-        RUN = self.task_config.RUN
-        cyc = str(self.task_config.cyc).zfill(2)
-        bcyc = str(self.task_config.MARINE_WINDOW_BEGIN.hour).zfill(2)
-        bdate = self.task_config.MARINE_WINDOW_BEGIN_ISO
-        mdate = self.task_config.MARINE_WINDOW_MIDDLE_ISO
-        nmem_ens = int(self.task_config.NMEM_ENS)
-
-        logger.info(f"---------------- Copy from RUNDIR to COMOUT")
-
-        post_file_list = []
-
-        # Make a copy the IAU increment
-        post_file_list.append([os.path.join(anl_dir, 'ocn.inc.nc'),
-                               os.path.join(com_ocean_analysis, f'{RUN}.t{cyc}z.ocninc.nc')])
-
-        domains = ['ocn', 'ice']
-        for domain in domains:
-            '''
-            # Copy of the diagonal of the background error for the cycle
-            post_file_list.append([os.path.join(anl_dir, f'{domain}.bkgerr_stddev.incr.{mdate}.nc'),
-                                   os.path.join(com_ocean_analysis, f'{RUN}.t{cyc}z.{domain}.bkgerr_stddev.nc')])
-
-            # Copy the recentering error
-            if nmem_ens > 2:
-                post_file_list.append([os.path.join(anl_dir, 'static_ens', f'{domain}.ssh_recentering_error.incr.{bdate}.nc'),
-                                       os.path.join(com_ocean_analysis, f'{RUN}.t{cyc}z.{domain}.recentering_error.nc')])
-            '''
-
-            # Copy the ice and ocean increments
-            post_file_list.append([os.path.join(anl_dir, 'Data', f'{domain}.3dvarfgat_pseudo.incr.{mdate}.nc'),
-                                   os.path.join(com_ocean_analysis, f'{RUN}.t{cyc}z.{domain}.incr.nc')])
-
-            # Copy the analysis at the start of the window
-            post_file_list.append([os.path.join(anl_dir, 'Data', f'{domain}.3dvarfgat_pseudo.an.{mdate}.nc'),
-                                   os.path.join(com_ocean_analysis, f'{RUN}.t{cyc}z.{domain}ana.nc')])
-
-        # Copy soca2cice ice increment
-        post_file_list.append([os.path.join(anl_dir, 'Data', f'ice.soca2cice.incr.{bdate}.nc'),
-                              os.path.join(com_ocean_analysis, f'{RUN}.t{cyc}z.ice.incr.postproc.nc')])
-
-        # Copy of the ssh diagnostics
-        if nmem_ens > 2:
-            for string in ['ssh_steric_stddev', 'ssh_unbal_stddev', 'ssh_total_stddev', 'steric_explained_variance']:
-                post_file_list.append([os.path.join(anl_dir, 'staticb', f'ocn.{string}.incr.{bdate}.nc'),
-                                       os.path.join(com_ocean_analysis, f'{RUN}.t{cyc}z.ocn.{string}.nc')])
-
-        # Copy DA grid (computed for the start of the window)
-        post_file_list.append([os.path.join(anl_dir, 'soca_gridspec.nc'),
-                               os.path.join(com_ocean_analysis, f'{RUN}.t{bcyc}z.ocngrid.nc')])
-
-        # Copy the CICE analysis restart
-        if os.getenv('DOIAU') == "YES":
-            cice_rst_date = self.task_config.MARINE_WINDOW_BEGIN.strftime('%Y%m%d.%H%M%S')
-        else:
-            cice_rst_date = cdate.strftime('%Y%m%d.%H%M%S')
+        self.jedi_dict[jedi_dict_key].execute()
 
-        post_file_list.append([os.path.join(anl_dir, 'Data', f'{cice_rst_date}.cice_model.res.nc'),
-                               os.path.join(com_ice_analysis, f'{cice_rst_date}.cice_model_anl.res.nc')])
+    @logit(logger)
+    def finalize(self: Task) -> None:
+        """Finalize a global marine analysis
 
-        FileHandler({'copy': post_file_list}).sync()
+        This method will finalize a global marine analysis.
+        This includes:
+        - Saving output files to COM
+        - Saving observation statistics to COM
 
-        # create COM sub-directories
-        FileHandler({'mkdir': [os.path.join(com_ocean_analysis, 'diags'),
-                               os.path.join(com_ocean_analysis, 'bump'),
-                               os.path.join(com_ocean_analysis, 'yaml')]}).sync()
+        Parameters
+        ----------
+        None
 
-        # ioda output files
-        fh_list = list_all_files(os.path.join(anl_dir, 'diags'),
-                                 os.path.join(com_ocean_analysis, 'diags'))
+        Returns
+        ----------
+        None
+        """
 
-        # yaml configurations
-        fh_list = list_all_files(os.path.join(anl_dir),
-                                 os.path.join(com_ocean_analysis, 'yaml'), wc='*.yaml', fh_list=fh_list)
+        # Save output files to COM
+        logger.info(f"Copy files from {self.task_config.DATA} to {self.task_config.COMOUT_OCEAN_ANALYSIS}")
+        soca_finalize_list = parse_j2yaml(self.task_config.MARINE_DET_FINALIZE_YAML_TMPL, self.task_config)
+        FileHandler(soca_finalize_list).sync()
 
-        FileHandler({'copy': fh_list}).sync()
+        # Save obs diag statistics to COM
+        diags_list = self.jedi_dict['soca_diag_stats'].render_jcb(self.task_config, 'soca_diags_finalize')
+        FileHandler(diags_list).sync()
 
     @logit(logger)
-    def obs_space_stats(self: Task) -> None:
-        """Observation space statistics
-           This method computes a few basic statistics on the observation spaces
+    def initialize_obs_stats(self) -> None:
+        """Initialize the observation statistics
+
+        This method will initialize the observation statistics
+        This includes:
+        - ...
+
+        Parameters
+        ----------
+        None
+
+        Returns
+        ----------
+        None
         """
 
-        # obs space statistics
-        logger.info(f"---------------- Compute basic stats")
-        diags_list = glob.glob(os.path.join(os.path.join(self.task_config.COMOUT_OCEAN_ANALYSIS, 'diags', '*.nc4')))
-        obsstats_j2yaml = str(os.path.join(self.task_config.PARMgfs, 'gdas', 'soca', 'obs', 'obs_stats.yaml.j2'))
-
-        # function to create a minimalist ioda obs sapce
-        def create_obs_space(data):
-            os_dict = {"obs space": {
-                       "name": data["obs_space"],
-                       "obsdatain": {
-                           "engine": {"type": "H5File", "obsfile": data["obsfile"]}
-                       },
-                       "simulated variables": [data["variable"]]
-                       },
-                       "variable": data["variable"],
-                       "experiment identifier": data["pslot"],
-                       "csv output": data["csv_output"]
-                       }
-            return os_dict
-
-        # get the experiment id
-        pslot = self.task_config.PSLOT
-
-        # iterate through the obs spaces and generate the yaml for gdassoca_obsstats.x
-        obs_spaces = []
-        for obsfile in diags_list:
-
-            # define an obs space name
-            obs_space = re.sub(r'\.\d{10}\.nc4$', '', os.path.basename(obsfile))
-
-            # get the variable name, assume 1 variable per file
-            nc = netCDF4.Dataset(obsfile, 'r')
-            variable = next(iter(nc.groups["ombg"].variables))
-            nc.close()
-
-            # filling values for the templated yaml
-            data = {'obs_space': os.path.basename(obsfile),
-                    'obsfile': obsfile,
-                    'pslot': pslot,
-                    'variable': variable,
-                    'csv_output': os.path.join(self.task_config.COMOUT_OCEAN_ANALYSIS,
-                                               f"{self.task_config.OPREFIX}ocn.{obs_space}.stats.csv")}
-            obs_spaces.append(create_obs_space(data))
-
-        # create the yaml
-        data = {'obs_spaces': obs_spaces}
-        conf = parse_j2yaml(path=obsstats_j2yaml, data=data)
-        stats_yaml = 'diag_stats.yaml'
-        conf.save(stats_yaml)
-
-        # run the application
-        mdau.link_executable(self.task_config, 'gdassoca_obsstats.x')
-        command = f"{os.getenv('launcher')} -n 1"
-        exec_cmd = Executable(command)
-        exec_name = os.path.join(self.task_config.DATA, 'gdassoca_obsstats.x')
-        exec_cmd.add_default_arg(exec_name)
-        exec_cmd.add_default_arg(stats_yaml)
-
-        mdau.run(exec_cmd)
+        #
+        cleaned_observations = []
+        obs_variables = {}
+        for obs_space in self.jedi_dict['var'].jedi_config.input_config['cost function']['observations']['observers']:
+            name = obs_space['obs space']['name']
+            variable = obs_space['obs space']['simulated variables'][0]
+
+            cleaned_observations.append(name)
+            obs_variables[name] = variable
+
+        # Update the task_config with the observation variables
+        self.task_config['cleaned_observations'] = cleaned_observations
+        self.task_config['obs_variables'] = obs_variables
+
+        # Initialize the observation statistics
+        logger.info(f"Initializing JEDI SOCA observation statistics application")
+        self.jedi_dict['soca_diag_stats'].initialize(self.task_config)
diff --git a/ush/python/pygfs/task/marine_bmat.py b/ush/python/pygfs/task/marine_bmat.py
index 11bc731723a..b2e30caa5a2 100644
--- a/ush/python/pygfs/task/marine_bmat.py
+++ b/ush/python/pygfs/task/marine_bmat.py
@@ -75,7 +75,7 @@ def __init__(self, config):
         # Create dictionary of Jedi objects
         expected_keys = ['gridgen', 'soca_diagb', 'soca_parameters_diffusion_vt', 'soca_setcorscales',
                          'soca_parameters_diffusion_hz', 'soca_ensb', 'soca_ensweights', 'soca_chgres']
-        self.jedi_dict = Jedi.get_jedi_dict(self.task_config.JEDI_CONFIG_YAML, self.task_config, expected_keys)
+        self.jedi_dict = Jedi.get_jedi_dict(self.task_config.JEDI_CONFIG_YAML_BMAT, self.task_config, expected_keys)
 
     @logit(logger)
     def initialize(self: Task) -> None:
diff --git a/ush/python/pygfs/task/marine_letkf.py b/ush/python/pygfs/task/marine_letkf.py
index b802d964f8e..12bd5f3cdc8 100644
--- a/ush/python/pygfs/task/marine_letkf.py
+++ b/ush/python/pygfs/task/marine_letkf.py
@@ -55,8 +55,6 @@ def __init__(self, config: Dict) -> None:
         self.task_config.obs_dir = os.path.join(self.task_config.DATA, 'obs')
         self.task_config.ENSPERT_RELPATH = _enspert_relpath
         self.task_config.PARMsoca = os.path.join(self.task_config.PARMgfs, 'gdas', 'soca')
-        self.task_config.cyc = os.getenv('cyc')
-        self.task_config.PDY = os.getenv('PDY')
         self.task_config.app_path_observations = self.task_config.MARINE_JCB_GDAS_OBS
         self.task_config.letkf_app = "true"
 
@@ -105,7 +103,7 @@ def initialize(self):
         for key in keys:
             stageconfig[key] = self.task_config[key]
 
-        jcb_base_yaml = os.path.join(self.task_config.PARMsoca, 'marine-jcb-base.yaml')
+        jcb_base_yaml = os.path.join(self.task_config.PARMsoca, 'jcb-base.yaml.j2')
         jcb_base_config = parse_j2yaml(path=jcb_base_yaml, data=stageconfig)
 
         jcb_config = {**jcb_base_config, **stageconfig}
diff --git a/ush/python/pygfs/task/snow_analysis.py b/ush/python/pygfs/task/snow_analysis.py
index 9dbda9406fa..4b4e71a541a 100644
--- a/ush/python/pygfs/task/snow_analysis.py
+++ b/ush/python/pygfs/task/snow_analysis.py
@@ -102,9 +102,6 @@ def initialize(self) -> None:
         ----------
         None
         """
-        # initialize JEDI variational application
-        logger.info(f"Initializing JEDI variational DA application")
-        self.jedi_dict['snowanlvar'].initialize(self.task_config)
 
         # stage backgrounds
         logger.info(f"Staging background files from {self.task_config.VAR_BKG_STAGING_YAML}")
@@ -143,6 +140,10 @@ def initialize(self) -> None:
         ]
         FileHandler({'mkdir': newdirs}).sync()
 
+        # initialize JEDI variational application
+        logger.info(f"Initializing JEDI variational DA application")
+        self.jedi_dict['snowanlvar'].initialize(self.task_config, clean_empty_obsspaces=True)
+
     @logit(logger)
     def prepare_IMS(self) -> None:
         """Prepare the IMS data for a global snow analysis
diff --git a/ush/python/pygfs/task/snowens_analysis.py b/ush/python/pygfs/task/snowens_analysis.py
index 8f4c9890caf..b817f2f1900 100644
--- a/ush/python/pygfs/task/snowens_analysis.py
+++ b/ush/python/pygfs/task/snowens_analysis.py
@@ -103,13 +103,6 @@ def initialize(self) -> None:
         ----------
         None
         """
-        # initialize JEDI variational application
-        logger.info(f"Initializing JEDI variational DA application")
-        self.jedi_dict['snowanlvar'].initialize(self.task_config)
-
-        # initialize ensemble mean computation
-        logger.info(f"Initializing JEDI ensemble mean application")
-        self.jedi_dict['esnowanlensmean'].initialize(self.task_config)
 
         # stage backgrounds
         logger.info(f"Staging background files from {self.task_config.SNOW_ENS_STAGE_TMPL}")
@@ -165,6 +158,14 @@ def initialize(self) -> None:
         ]
         FileHandler({'mkdir': newdirs}).sync()
 
+        # initialize JEDI variational application
+        logger.info(f"Initializing JEDI variational DA application")
+        self.jedi_dict['snowanlvar'].initialize(self.task_config, clean_empty_obsspaces=True)
+
+        # initialize ensemble mean computation
+        logger.info(f"Initializing JEDI ensemble mean application")
+        self.jedi_dict['esnowanlensmean'].initialize(self.task_config)
+
     @logit(logger)
     def prepare_IMS(self) -> None:
         """Prepare the IMS data for a global snow analysis
diff --git a/ush/python/pygfs/utils/marine_da_utils.py b/ush/python/pygfs/utils/marine_da_utils.py
index 748b5ff9c9a..ebb523a60b6 100644
--- a/ush/python/pygfs/utils/marine_da_utils.py
+++ b/ush/python/pygfs/utils/marine_da_utils.py
@@ -98,9 +98,9 @@ def test_hist_date(histfile: str, ref_date: datetime) -> None:
 
 
 @logit(logger)
-def gen_bkg_list(bkg_path: str, window_begin=' ', yaml_name='bkg.yaml', ice_rst=False) -> None:
+def gen_bkg_list(bkg_path: str, window_begin=' ', ice_rst=False) -> None:
     """
-    Generate a YAML of the list of backgrounds for the pseudo model
+    Generate a list of backgrounds for the pseudo model
     """
 
     # Pseudo model parameters (time step, start date)
@@ -109,8 +109,6 @@ def gen_bkg_list(bkg_path: str, window_begin=' ', yaml_name='bkg.yaml', ice_rst=
     bkg_date = window_begin
 
     # Construct list of background file names
-    cyc = str(os.getenv('cyc')).zfill(2)
-    gcyc = str((int(cyc) - 6) % 24).zfill(2)  # previous cycle
     fcst_hrs = list(range(6, 10, dt_pseudo))
     files = []
     for fcst_hr in fcst_hrs:
@@ -118,15 +116,13 @@ def gen_bkg_list(bkg_path: str, window_begin=' ', yaml_name='bkg.yaml', ice_rst=
 
     # Identify the ocean background that will be used for the  vertical coordinate remapping
     ocn_filename_ic = './INPUT/MOM.res.nc'
-    test_hist_date(ocn_filename_ic, bkg_date)  # assert date of the history file is correct
 
     # Copy/process backgrounds and generate background yaml list
     bkg_list = []
     for bkg in files:
         logger.info(f"****************** bkg: {bkg}")
-        # assert validity of the ocean bkg date, remove basename
+        # remove basename of ocean bkg
         bkg_date = bkg_date + timedelta(hours=dt_pseudo)
-        test_hist_date(bkg, bkg_date)
         ocn_filename = os.path.splitext(os.path.basename(bkg))[0] + '.nc'
 
         # prepare the seaice background, aggregate if the backgrounds are CICE restarts
@@ -141,8 +137,7 @@ def gen_bkg_list(bkg_path: str, window_begin=' ', yaml_name='bkg.yaml', ice_rst=
 
         bkg_list.append(bkg_dict)
 
-    # save pseudo model yaml configuration
-    save_as_yaml(bkg_list, yaml_name)
+    return bkg_list
 
 
 @logit(logger)


Reviews:
user: login: guillaumevernieres
body: Looks good, except that we do need to check the date of the backgrounds.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T12:54:27Z
user: login: DavidNew-NOAA
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T13:00:12Z
user: login: guillaumevernieres
body: Looks good. Thanks @DavidNew-NOAA .
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T12:06:15Z
user: login: shlyaeva
body: Impressive how much shorter it is now! Not an expert review from me, but very helpful for me to see.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T21:43:14Z
user: login: AndrewEichmann-NOAA
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T13:22:19Z
user: login: AndrewEichmann-NOAA
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T13:25:01Z
user: login: AndrewEichmann-NOAA
body: Looks slick
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T13:25:38Z
user: login: aerorahul
body: Looks good.  Just 2 suggestions.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T13:44:04Z
user: login: aerorahul
body: lgtm
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-05-15T17:58:40Z
user: login: shlyaeva
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T15:15:50Z
user: login: CoryMartin-NOAA
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T14:44:06Z
user: login: DavidNew-NOAA
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T14:50:28Z
user: login: CoryMartin-NOAA
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T14:51:57Z
user: login: CoryMartin-NOAA
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T12:37:40Z
user: login: aerorahul
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-23T12:45:33Z
user: login: aerorahul
body: lgtm
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-23T12:46:07Z

Review comments:
diff_hunk: @@ -109,24 +109,20 @@ def gen_bkg_list(bkg_path: str, window_begin=' ', yaml_name='bkg.yaml', ice_rst=
     bkg_date = window_begin
 
     # Construct list of background file names
-    cyc = str(os.getenv('cyc')).zfill(2)
-    gcyc = str((int(cyc) - 6) % 24).zfill(2)  # previous cycle
     fcst_hrs = list(range(6, 10, dt_pseudo))
     files = []
     for fcst_hr in fcst_hrs:
         files.append(os.path.join(bkg_path, f"ocean.bkg.f{str(fcst_hr).zfill(3)}.nc"))
 
     # Identify the ocean background that will be used for the  vertical coordinate remapping
     ocn_filename_ic = './INPUT/MOM.res.nc'
-    test_hist_date(ocn_filename_ic, bkg_date)  # assert date of the history file is correct
 
     # Copy/process backgrounds and generate background yaml list
     bkg_list = []
     for bkg in files:
         logger.info(f"****************** bkg: {bkg}")
-        # assert validity of the ocean bkg date, remove basename
+        # remove basename of ocean bkg
         bkg_date = bkg_date + timedelta(hours=dt_pseudo)
-        test_hist_date(bkg, bkg_date)
path: ush/python/pygfs/utils/marine_da_utils.py
user: login: guillaumevernieres
body: You can't remove this @DavidNew-NOAA .
created_at: 2025-05-05T12:52:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -109,24 +109,20 @@ def gen_bkg_list(bkg_path: str, window_begin=' ', yaml_name='bkg.yaml', ice_rst=
     bkg_date = window_begin
 
     # Construct list of background file names
-    cyc = str(os.getenv('cyc')).zfill(2)
-    gcyc = str((int(cyc) - 6) % 24).zfill(2)  # previous cycle
     fcst_hrs = list(range(6, 10, dt_pseudo))
     files = []
     for fcst_hr in fcst_hrs:
         files.append(os.path.join(bkg_path, f"ocean.bkg.f{str(fcst_hr).zfill(3)}.nc"))
 
     # Identify the ocean background that will be used for the  vertical coordinate remapping
     ocn_filename_ic = './INPUT/MOM.res.nc'
-    test_hist_date(ocn_filename_ic, bkg_date)  # assert date of the history file is correct
 
     # Copy/process backgrounds and generate background yaml list
     bkg_list = []
     for bkg in files:
         logger.info(f"****************** bkg: {bkg}")
-        # assert validity of the ocean bkg date, remove basename
+        # remove basename of ocean bkg
         bkg_date = bkg_date + timedelta(hours=dt_pseudo)
-        test_hist_date(bkg, bkg_date)
path: ush/python/pygfs/utils/marine_da_utils.py
user: login: DavidNew-NOAA
body: @guillaumevernieres It's actually moved rather than re-moved. I put the date testing in the 'initialize()' method. I'll explain why when we tag up.
created_at: 2025-05-05T13:00:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,452 +48,151 @@ def __init__(self, config):
         else:
             _enspert_relpath = None
 
+        # Determine background error model
+        if self.task_config.NMEM_ENS >= 2:
+            _berror_model = 'marine_background_error_hybrid_diffusion_diffusion'
+        else:
+            _berror_model = 'marine_background_error_static_diffusion'
+
         # Create a local dictionary that is repeatedly used across this class
         local_dict = AttrDict(
             {
                 'PARMsoca': os.path.join(self.task_config.PARMgfs, 'gdas', 'soca'),
                 'MARINE_WINDOW_BEGIN': _window_begin,
-                'MARINE_WINDOW_BEGIN_ISO': _window_begin.strftime('%Y-%m-%dT%H:%M:%SZ'),
                 'MARINE_WINDOW_END': _window_end,
-                'MARINE_WINDOW_LENGTH': f"PT{self.task_config['assim_freq']}H",
                 'MARINE_WINDOW_MIDDLE': self.task_config.current_cycle,
-                'MARINE_WINDOW_MIDDLE_ISO': self.task_config.current_cycle.strftime('%Y-%m-%dT%H:%M:%SZ'),
+                'MARINE_WINDOW_LENGTH': f"PT{self.task_config['assim_freq']}H",
+                'MARINE_WINDOW_BEGIN_ISO': to_isotime(_window_begin),
+                'MARINE_WINDOW_MIDDLE_ISO': to_isotime(self.task_config.current_cycle),
                 'ENSPERT_RELPATH': _enspert_relpath,
                 'CALC_SCALE_EXEC': _calc_scale_exec,
                 'OPREFIX': f"{self.task_config.RUN}.t{self.task_config.cyc:02d}z.",
                 'APREFIX': f"{self.task_config.RUN}.t{self.task_config.cyc:02d}z.",
-                'app_path_observations': self.task_config.MARINE_JCB_GDAS_OBS
+                'berror_model': _berror_model,
+                'MOM6_LEVS': mdau.get_mom6_levels(str(self.task_config.OCNRES).zfill(3)),
+                'app_path_observations': self.task_config.MARINE_JCB_GDAS_OBS,
+                'marine_pseudo_model_states': mdau.gen_bkg_list(bkg_path='./bkg',
+                                                                window_begin=_window_begin)
             }
         )
 
         # Extend task_config with local_dict
         self.task_config.update(local_dict)
 
+        # Construct dictionary of JEDI objects, one for each JEDI application need for the analysis
+        expected_keys = ['var', 'socaincr2mom6', 'soca_2cice_global', 'soca_diag_stats']
+        self.jedi_dict = Jedi.get_jedi_dict(self.task_config.JEDI_CONFIG_YAML_ANALYSIS, self.task_config, expected_keys)
+
     @logit(logger)
     def initialize(self: Task) -> None:
-        """Initialize the marine analysis
+        """Initialize the marine analysis task
 
         This method will initialize the marine analysis.
         This includes:
-        - staging the deterministic backgrounds (middle of window)
         - staging SOCA fix files
-        - staging static ensemble members (optional)
-        - staging ensemble members (optional)
-        - generating the YAML files for the JEDI and GDASApp executables
-        - creating output directories
-        """
-        super().initialize()
-
-        # prepare the directory structure to run SOCA
-        self._prep_scratch_dir()
-
-        # fetch observations from COMROOT
-        # TODO(G.V. or A.E.): Keep a copy of the obs in the scratch fs after the obs prep job
-        self._fetch_observations()
-
-        # stage the ocean and ice backgrounds for FGAT
-        bkg_list = parse_j2yaml(self.task_config.MARINE_DET_STAGE_BKG_YAML_TMPL, self.task_config)
-        FileHandler(bkg_list).sync()
-
-        # stage the soca grid
-        FileHandler({'copy': [[os.path.join(self.task_config.COMIN_OCEAN_BMATRIX, 'soca_gridspec.nc'),
-                               os.path.join(self.task_config.DATA, 'soca_gridspec.nc')]]}).sync()
-
-        # link the flow dependent static B resources from the B-matrix task of the same cycle
-        os.symlink('../staticb', 'staticb')
-
-        # hybrid EnVAR case
-        if self.task_config.DOHYBVAR_OCN == "YES" or self.task_config.NMEM_ENS >= 2:
-            # stage the ensemble weights
-            logger.debug(f"Stage ensemble weights for the hybrid background error")
-            FileHandler({'copy': [[os.path.join(self.task_config.COMIN_OCEAN_BMATRIX, f'{self.task_config.APREFIX}ocean.ens_weights.nc'),
-                                   os.path.join(self.task_config.DATA, 'ocean.ens_weights.nc')],
-                                  [os.path.join(self.task_config.COMIN_ICE_BMATRIX, f'{self.task_config.APREFIX}ice.ens_weights.nc'),
-                                   os.path.join(self.task_config.DATA, 'ice.ens_weights.nc')]]}).sync()
-
-        # prepare the yaml configuration to run the SOCA variational application
-        self._prep_variational_yaml()
-
-        # prepare the yaml configuration to run the SOCA to MOM6 IAU increment
-        self._prep_checkpoint()
-
-    @logit(logger)
-    def _fetch_observations(self: Task) -> None:
-        """Fetch observations from COMIN_OBS
-
-        This method will fetch the observations for the cycle and check the
-        list against what is available for the cycle.
-        """
-
-        # get the list of observations
-
-        # "observations" is expected by later JCB code to populate it with config info,
-        # but the obs_list as such is needed later
-        self.task_config.observations = parse_j2yaml(self.task_config.MARINE_OBS_LIST_YAML, self.task_config)['observations']
-        self.task_config.obs_list = self.task_config.observations
-
-        obsconfigfile = os.path.join(self.task_config['PARMgfs'], 'gdas/soca/obs/obs_list_base_yaml.j2')
-        self.task_config.observations = parse_j2yaml(obsconfigfile, self.task_config)
-
-        obs_files = []
-
-        for observer in self.task_config['observations']['observers']:
-            filename = f"{self.task_config.OPREFIX}{observer['obs space']['name'].lower()}.{to_YMD(self.task_config.PDY)}{self.task_config.cyc:02d}.nc4"
-            logger.info(f"******** {filename}")
-            obs_files.append(filename)
-
-        obs_files_to_copy = []
-
-        # copy obs from COM_OBS to DATA/obs
-        for obs_file in obs_files:
-            logger.info(f"******* {obs_file}")
-            obs_src = os.path.join(self.task_config.COM_OBS, obs_file)
-            obs_dst = os.path.join(self.task_config.DATA, 'obs', obs_file)
-            logger.info(f"******* {obs_src}")
-            if os.path.exists(obs_src):
-                logger.info(f"******* fetching {obs_file}")
-                obs_files_to_copy.append([obs_src, obs_dst])
-            else:
-                logger.info(f"******* {obs_file} is not in the database")
+        - preparing the namelists for deterministic MOM6 and analysis geometry
+        - staging observations
+        - staging input YAMLs for SOCA utilities
+        - staging the deterministic backgrounds (middle of window)
+        - staging files and link directories from B-matrix job needed for deterministic analysis
+        - generating list of model pseudo states
+        - initializing all the JEDI applications required for the marine analysis
 
-        FileHandler({'copy': obs_files_to_copy}).sync()
+        Parameters
+        ----------
+        None
 
-    @logit(logger)
-    def _prep_scratch_dir(self: Task) -> None:
-        """Create and stage all the resources needed to run SOCA/JEDI, including the necesssary
-           directory structure to run the SOCA variational application
+        Returns
+        ----------
+        None
         """
-        logger.info(f"---------------- Setup runtime environement")
-
-        anl_dir = self.task_config.DATA
-
-        # create analysis directories
-        diags = os.path.join(anl_dir, 'diags')            # output dir for soca DA obs space
-        obs_in = os.path.join(anl_dir, 'obs')             # input      "           "
-        anl_out = os.path.join(anl_dir, 'Data')           # output dir for soca DA
-        FileHandler({'mkdir': [diags, obs_in, anl_out]}).sync()
 
         # stage fix files
         logger.info(f"Staging SOCA fix files from {self.task_config.SOCA_INPUT_FIX_DIR}")
         soca_fix_list = parse_j2yaml(self.task_config.SOCA_FIX_YAML_TMPL, self.task_config)
         FileHandler(soca_fix_list).sync()
 
         # prepare the deterministic MOM6 input.nml
+        logger.info(f"Preparing deterministic MOM6 input namelist")
         mdau.prep_input_nml(self.task_config)
 
         # prepare the input.nml for the analysis geometry
+        logger.info(f"Preparing analysis geometry input namelist")
         mdau.prep_input_nml(self.task_config, output_nml="./anl_geom/mom_input.nml",
                             simple_geom=True, mom_input="./anl_geom/MOM_input")
 
+        # fetch observations from COMROOT
+        # TODO(G.V. or A.E.): Keep a copy of the obs in the scratch fs after the obs prep job
+        logger.info(f"Staging observations from {self.task_config.COM_OBS}")
+        obs_list = self.jedi_dict['var'].render_jcb(self.task_config, 'soca_obs_staging')
path: ush/python/pygfs/task/marine_analysis.py
user: login: AndrewEichmann-NOAA
body: Curious what `soca_obs_staging` is referencing - it doesn't show up in any repos
created_at: 2025-05-14T13:22:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,452 +48,151 @@ def __init__(self, config):
         else:
             _enspert_relpath = None
 
+        # Determine background error model
+        if self.task_config.NMEM_ENS >= 2:
+            _berror_model = 'marine_background_error_hybrid_diffusion_diffusion'
+        else:
+            _berror_model = 'marine_background_error_static_diffusion'
+
         # Create a local dictionary that is repeatedly used across this class
         local_dict = AttrDict(
             {
                 'PARMsoca': os.path.join(self.task_config.PARMgfs, 'gdas', 'soca'),
                 'MARINE_WINDOW_BEGIN': _window_begin,
-                'MARINE_WINDOW_BEGIN_ISO': _window_begin.strftime('%Y-%m-%dT%H:%M:%SZ'),
                 'MARINE_WINDOW_END': _window_end,
-                'MARINE_WINDOW_LENGTH': f"PT{self.task_config['assim_freq']}H",
                 'MARINE_WINDOW_MIDDLE': self.task_config.current_cycle,
-                'MARINE_WINDOW_MIDDLE_ISO': self.task_config.current_cycle.strftime('%Y-%m-%dT%H:%M:%SZ'),
+                'MARINE_WINDOW_LENGTH': f"PT{self.task_config['assim_freq']}H",
+                'MARINE_WINDOW_BEGIN_ISO': to_isotime(_window_begin),
+                'MARINE_WINDOW_MIDDLE_ISO': to_isotime(self.task_config.current_cycle),
                 'ENSPERT_RELPATH': _enspert_relpath,
                 'CALC_SCALE_EXEC': _calc_scale_exec,
                 'OPREFIX': f"{self.task_config.RUN}.t{self.task_config.cyc:02d}z.",
                 'APREFIX': f"{self.task_config.RUN}.t{self.task_config.cyc:02d}z.",
-                'app_path_observations': self.task_config.MARINE_JCB_GDAS_OBS
+                'berror_model': _berror_model,
+                'MOM6_LEVS': mdau.get_mom6_levels(str(self.task_config.OCNRES).zfill(3)),
+                'app_path_observations': self.task_config.MARINE_JCB_GDAS_OBS,
+                'marine_pseudo_model_states': mdau.gen_bkg_list(bkg_path='./bkg',
+                                                                window_begin=_window_begin)
             }
         )
 
         # Extend task_config with local_dict
         self.task_config.update(local_dict)
 
+        # Construct dictionary of JEDI objects, one for each JEDI application need for the analysis
+        expected_keys = ['var', 'socaincr2mom6', 'soca_2cice_global', 'soca_diag_stats']
+        self.jedi_dict = Jedi.get_jedi_dict(self.task_config.JEDI_CONFIG_YAML_ANALYSIS, self.task_config, expected_keys)
+
     @logit(logger)
     def initialize(self: Task) -> None:
-        """Initialize the marine analysis
+        """Initialize the marine analysis task
 
         This method will initialize the marine analysis.
         This includes:
-        - staging the deterministic backgrounds (middle of window)
         - staging SOCA fix files
-        - staging static ensemble members (optional)
-        - staging ensemble members (optional)
-        - generating the YAML files for the JEDI and GDASApp executables
-        - creating output directories
-        """
-        super().initialize()
-
-        # prepare the directory structure to run SOCA
-        self._prep_scratch_dir()
-
-        # fetch observations from COMROOT
-        # TODO(G.V. or A.E.): Keep a copy of the obs in the scratch fs after the obs prep job
-        self._fetch_observations()
-
-        # stage the ocean and ice backgrounds for FGAT
-        bkg_list = parse_j2yaml(self.task_config.MARINE_DET_STAGE_BKG_YAML_TMPL, self.task_config)
-        FileHandler(bkg_list).sync()
-
-        # stage the soca grid
-        FileHandler({'copy': [[os.path.join(self.task_config.COMIN_OCEAN_BMATRIX, 'soca_gridspec.nc'),
-                               os.path.join(self.task_config.DATA, 'soca_gridspec.nc')]]}).sync()
-
-        # link the flow dependent static B resources from the B-matrix task of the same cycle
-        os.symlink('../staticb', 'staticb')
-
-        # hybrid EnVAR case
-        if self.task_config.DOHYBVAR_OCN == "YES" or self.task_config.NMEM_ENS >= 2:
-            # stage the ensemble weights
-            logger.debug(f"Stage ensemble weights for the hybrid background error")
-            FileHandler({'copy': [[os.path.join(self.task_config.COMIN_OCEAN_BMATRIX, f'{self.task_config.APREFIX}ocean.ens_weights.nc'),
-                                   os.path.join(self.task_config.DATA, 'ocean.ens_weights.nc')],
-                                  [os.path.join(self.task_config.COMIN_ICE_BMATRIX, f'{self.task_config.APREFIX}ice.ens_weights.nc'),
-                                   os.path.join(self.task_config.DATA, 'ice.ens_weights.nc')]]}).sync()
-
-        # prepare the yaml configuration to run the SOCA variational application
-        self._prep_variational_yaml()
-
-        # prepare the yaml configuration to run the SOCA to MOM6 IAU increment
-        self._prep_checkpoint()
-
-    @logit(logger)
-    def _fetch_observations(self: Task) -> None:
-        """Fetch observations from COMIN_OBS
-
-        This method will fetch the observations for the cycle and check the
-        list against what is available for the cycle.
-        """
-
-        # get the list of observations
-
-        # "observations" is expected by later JCB code to populate it with config info,
-        # but the obs_list as such is needed later
-        self.task_config.observations = parse_j2yaml(self.task_config.MARINE_OBS_LIST_YAML, self.task_config)['observations']
-        self.task_config.obs_list = self.task_config.observations
-
-        obsconfigfile = os.path.join(self.task_config['PARMgfs'], 'gdas/soca/obs/obs_list_base_yaml.j2')
-        self.task_config.observations = parse_j2yaml(obsconfigfile, self.task_config)
-
-        obs_files = []
-
-        for observer in self.task_config['observations']['observers']:
-            filename = f"{self.task_config.OPREFIX}{observer['obs space']['name'].lower()}.{to_YMD(self.task_config.PDY)}{self.task_config.cyc:02d}.nc4"
-            logger.info(f"******** {filename}")
-            obs_files.append(filename)
-
-        obs_files_to_copy = []
-
-        # copy obs from COM_OBS to DATA/obs
-        for obs_file in obs_files:
-            logger.info(f"******* {obs_file}")
-            obs_src = os.path.join(self.task_config.COM_OBS, obs_file)
-            obs_dst = os.path.join(self.task_config.DATA, 'obs', obs_file)
-            logger.info(f"******* {obs_src}")
-            if os.path.exists(obs_src):
-                logger.info(f"******* fetching {obs_file}")
-                obs_files_to_copy.append([obs_src, obs_dst])
-            else:
-                logger.info(f"******* {obs_file} is not in the database")
+        - preparing the namelists for deterministic MOM6 and analysis geometry
+        - staging observations
+        - staging input YAMLs for SOCA utilities
+        - staging the deterministic backgrounds (middle of window)
+        - staging files and link directories from B-matrix job needed for deterministic analysis
+        - generating list of model pseudo states
+        - initializing all the JEDI applications required for the marine analysis
 
-        FileHandler({'copy': obs_files_to_copy}).sync()
+        Parameters
+        ----------
+        None
 
-    @logit(logger)
-    def _prep_scratch_dir(self: Task) -> None:
-        """Create and stage all the resources needed to run SOCA/JEDI, including the necesssary
-           directory structure to run the SOCA variational application
+        Returns
+        ----------
+        None
         """
-        logger.info(f"---------------- Setup runtime environement")
-
-        anl_dir = self.task_config.DATA
-
-        # create analysis directories
-        diags = os.path.join(anl_dir, 'diags')            # output dir for soca DA obs space
-        obs_in = os.path.join(anl_dir, 'obs')             # input      "           "
-        anl_out = os.path.join(anl_dir, 'Data')           # output dir for soca DA
-        FileHandler({'mkdir': [diags, obs_in, anl_out]}).sync()
 
         # stage fix files
         logger.info(f"Staging SOCA fix files from {self.task_config.SOCA_INPUT_FIX_DIR}")
         soca_fix_list = parse_j2yaml(self.task_config.SOCA_FIX_YAML_TMPL, self.task_config)
         FileHandler(soca_fix_list).sync()
 
         # prepare the deterministic MOM6 input.nml
+        logger.info(f"Preparing deterministic MOM6 input namelist")
         mdau.prep_input_nml(self.task_config)
 
         # prepare the input.nml for the analysis geometry
+        logger.info(f"Preparing analysis geometry input namelist")
         mdau.prep_input_nml(self.task_config, output_nml="./anl_geom/mom_input.nml",
                             simple_geom=True, mom_input="./anl_geom/MOM_input")
 
+        # fetch observations from COMROOT
+        # TODO(G.V. or A.E.): Keep a copy of the obs in the scratch fs after the obs prep job
+        logger.info(f"Staging observations from {self.task_config.COM_OBS}")
+        obs_list = self.jedi_dict['var'].render_jcb(self.task_config, 'soca_obs_staging')
path: ush/python/pygfs/task/marine_analysis.py
user: login: AndrewEichmann-NOAA
body: Ah, never mind, saw it in the` jcb-gdas` PR
created_at: 2025-05-14T13:25:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -257,6 +261,39 @@ def get_jedi_dict(jedi_config_yaml: str, task_config: AttrDict, expected_block_n
         # Return dictionary of JEDI objects
         return jedi_dict
 
+    @logit(logger)
+    def clean_empty_obsspaces(self):
+        """
+        Replace list of observers in JEDI input configuration dictionary with new list, removing
+        any observers with missing observation files.
+
+        Parameters
+        ----------
+        None
+
+        Returns
+        -------
+        None
+        """
+
+        # Get observers from JEDI input config
+        observers = find_value_in_nested_dict(self.jedi_config.input_config, 'observers')
+
+        # Check if observers list actually present
+        if observers:
+            # Create new list of observers
+            cleaned_observers = []
+            for obs_space in observers:
+                fname = obs_space['obs space']['obsdatain']['engine']['obsfile']
+                if os.path.isfile(fname):
+                    cleaned_observers.append(obs_space)
+                else:
+                    logger.info(f"WARNING: {fname} does not exist, removing obs space")
path: ush/python/pygfs/jedi/jedi.py
user: login: aerorahul
body: ```suggestion
                    logger.warning(f"WARNING: {fname} does not exist, removing obs space")
```
created_at: 2025-05-14T13:35:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -350,3 +387,75 @@ def extract_tar(tar_file: str) -> None:
     except tarfile.ExtractError as err:
         logger.exception(f"FATAL ERROR: unable to extract from {tar_file}")
         raise tarfile.ExtractError("FATAL ERROR: unable to extract from {tar_file}")
path: ush/python/pygfs/jedi/jedi.py
user: login: aerorahul
body: ```suggestion
        raise tarfile.ExtractError(f"FATAL ERROR: unable to extract from {tar_file}")
```
created_at: 2025-05-14T13:36:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,452 +48,151 @@ def __init__(self, config):
         else:
             _enspert_relpath = None
 
+        # Determine background error model
+        if self.task_config.NMEM_ENS >= 2:
+            _berror_model = 'marine_background_error_hybrid_diffusion_diffusion'
+        else:
+            _berror_model = 'marine_background_error_static_diffusion'
+
         # Create a local dictionary that is repeatedly used across this class
         local_dict = AttrDict(
             {
                 'PARMsoca': os.path.join(self.task_config.PARMgfs, 'gdas', 'soca'),
                 'MARINE_WINDOW_BEGIN': _window_begin,
-                'MARINE_WINDOW_BEGIN_ISO': _window_begin.strftime('%Y-%m-%dT%H:%M:%SZ'),
                 'MARINE_WINDOW_END': _window_end,
-                'MARINE_WINDOW_LENGTH': f"PT{self.task_config['assim_freq']}H",
                 'MARINE_WINDOW_MIDDLE': self.task_config.current_cycle,
-                'MARINE_WINDOW_MIDDLE_ISO': self.task_config.current_cycle.strftime('%Y-%m-%dT%H:%M:%SZ'),
+                'MARINE_WINDOW_LENGTH': f"PT{self.task_config['assim_freq']}H",
+                'MARINE_WINDOW_BEGIN_ISO': to_isotime(_window_begin),
+                'MARINE_WINDOW_MIDDLE_ISO': to_isotime(self.task_config.current_cycle),
                 'ENSPERT_RELPATH': _enspert_relpath,
                 'CALC_SCALE_EXEC': _calc_scale_exec,
                 'OPREFIX': f"{self.task_config.RUN}.t{self.task_config.cyc:02d}z.",
                 'APREFIX': f"{self.task_config.RUN}.t{self.task_config.cyc:02d}z.",
-                'app_path_observations': self.task_config.MARINE_JCB_GDAS_OBS
+                'berror_model': _berror_model,
+                'MOM6_LEVS': mdau.get_mom6_levels(str(self.task_config.OCNRES).zfill(3)),
+                'app_path_observations': self.task_config.MARINE_JCB_GDAS_OBS,
+                'marine_pseudo_model_states': mdau.gen_bkg_list(bkg_path='./bkg',
+                                                                window_begin=_window_begin)
             }
         )
 
         # Extend task_config with local_dict
         self.task_config.update(local_dict)
 
+        # Construct dictionary of JEDI objects, one for each JEDI application need for the analysis
+        expected_keys = ['var', 'socaincr2mom6', 'soca_2cice_global', 'soca_diag_stats']
+        self.jedi_dict = Jedi.get_jedi_dict(self.task_config.JEDI_CONFIG_YAML_ANALYSIS, self.task_config, expected_keys)
+
     @logit(logger)
     def initialize(self: Task) -> None:
-        """Initialize the marine analysis
+        """Initialize the marine analysis task
 
         This method will initialize the marine analysis.
         This includes:
-        - staging the deterministic backgrounds (middle of window)
         - staging SOCA fix files
-        - staging static ensemble members (optional)
-        - staging ensemble members (optional)
-        - generating the YAML files for the JEDI and GDASApp executables
-        - creating output directories
-        """
-        super().initialize()
-
-        # prepare the directory structure to run SOCA
-        self._prep_scratch_dir()
-
-        # fetch observations from COMROOT
-        # TODO(G.V. or A.E.): Keep a copy of the obs in the scratch fs after the obs prep job
-        self._fetch_observations()
-
-        # stage the ocean and ice backgrounds for FGAT
-        bkg_list = parse_j2yaml(self.task_config.MARINE_DET_STAGE_BKG_YAML_TMPL, self.task_config)
-        FileHandler(bkg_list).sync()
-
-        # stage the soca grid
-        FileHandler({'copy': [[os.path.join(self.task_config.COMIN_OCEAN_BMATRIX, 'soca_gridspec.nc'),
-                               os.path.join(self.task_config.DATA, 'soca_gridspec.nc')]]}).sync()
-
-        # link the flow dependent static B resources from the B-matrix task of the same cycle
-        os.symlink('../staticb', 'staticb')
-
-        # hybrid EnVAR case
-        if self.task_config.DOHYBVAR_OCN == "YES" or self.task_config.NMEM_ENS >= 2:
-            # stage the ensemble weights
-            logger.debug(f"Stage ensemble weights for the hybrid background error")
-            FileHandler({'copy': [[os.path.join(self.task_config.COMIN_OCEAN_BMATRIX, f'{self.task_config.APREFIX}ocean.ens_weights.nc'),
-                                   os.path.join(self.task_config.DATA, 'ocean.ens_weights.nc')],
-                                  [os.path.join(self.task_config.COMIN_ICE_BMATRIX, f'{self.task_config.APREFIX}ice.ens_weights.nc'),
-                                   os.path.join(self.task_config.DATA, 'ice.ens_weights.nc')]]}).sync()
-
-        # prepare the yaml configuration to run the SOCA variational application
-        self._prep_variational_yaml()
-
-        # prepare the yaml configuration to run the SOCA to MOM6 IAU increment
-        self._prep_checkpoint()
-
-    @logit(logger)
-    def _fetch_observations(self: Task) -> None:
-        """Fetch observations from COMIN_OBS
-
-        This method will fetch the observations for the cycle and check the
-        list against what is available for the cycle.
-        """
-
-        # get the list of observations
-
-        # "observations" is expected by later JCB code to populate it with config info,
-        # but the obs_list as such is needed later
-        self.task_config.observations = parse_j2yaml(self.task_config.MARINE_OBS_LIST_YAML, self.task_config)['observations']
-        self.task_config.obs_list = self.task_config.observations
-
-        obsconfigfile = os.path.join(self.task_config['PARMgfs'], 'gdas/soca/obs/obs_list_base_yaml.j2')
-        self.task_config.observations = parse_j2yaml(obsconfigfile, self.task_config)
-
-        obs_files = []
-
-        for observer in self.task_config['observations']['observers']:
-            filename = f"{self.task_config.OPREFIX}{observer['obs space']['name'].lower()}.{to_YMD(self.task_config.PDY)}{self.task_config.cyc:02d}.nc4"
-            logger.info(f"******** {filename}")
-            obs_files.append(filename)
-
-        obs_files_to_copy = []
-
-        # copy obs from COM_OBS to DATA/obs
-        for obs_file in obs_files:
-            logger.info(f"******* {obs_file}")
-            obs_src = os.path.join(self.task_config.COM_OBS, obs_file)
-            obs_dst = os.path.join(self.task_config.DATA, 'obs', obs_file)
-            logger.info(f"******* {obs_src}")
-            if os.path.exists(obs_src):
-                logger.info(f"******* fetching {obs_file}")
-                obs_files_to_copy.append([obs_src, obs_dst])
-            else:
-                logger.info(f"******* {obs_file} is not in the database")
+        - preparing the namelists for deterministic MOM6 and analysis geometry
+        - staging observations
+        - staging input YAMLs for SOCA utilities
+        - staging the deterministic backgrounds (middle of window)
+        - staging files and link directories from B-matrix job needed for deterministic analysis
+        - generating list of model pseudo states
+        - initializing all the JEDI applications required for the marine analysis
 
-        FileHandler({'copy': obs_files_to_copy}).sync()
+        Parameters
+        ----------
+        None
 
-    @logit(logger)
-    def _prep_scratch_dir(self: Task) -> None:
-        """Create and stage all the resources needed to run SOCA/JEDI, including the necesssary
-           directory structure to run the SOCA variational application
+        Returns
+        ----------
+        None
         """
-        logger.info(f"---------------- Setup runtime environement")
-
-        anl_dir = self.task_config.DATA
-
-        # create analysis directories
-        diags = os.path.join(anl_dir, 'diags')            # output dir for soca DA obs space
-        obs_in = os.path.join(anl_dir, 'obs')             # input      "           "
-        anl_out = os.path.join(anl_dir, 'Data')           # output dir for soca DA
-        FileHandler({'mkdir': [diags, obs_in, anl_out]}).sync()
 
         # stage fix files
         logger.info(f"Staging SOCA fix files from {self.task_config.SOCA_INPUT_FIX_DIR}")
         soca_fix_list = parse_j2yaml(self.task_config.SOCA_FIX_YAML_TMPL, self.task_config)
         FileHandler(soca_fix_list).sync()
 
         # prepare the deterministic MOM6 input.nml
+        logger.info(f"Preparing deterministic MOM6 input namelist")
         mdau.prep_input_nml(self.task_config)
 
         # prepare the input.nml for the analysis geometry
+        logger.info(f"Preparing analysis geometry input namelist")
         mdau.prep_input_nml(self.task_config, output_nml="./anl_geom/mom_input.nml",
                             simple_geom=True, mom_input="./anl_geom/MOM_input")
 
+        # fetch observations from COMROOT
+        # TODO(G.V. or A.E.): Keep a copy of the obs in the scratch fs after the obs prep job
+        logger.info(f"Staging observations from {self.task_config.COM_OBS}")
+        obs_list = self.jedi_dict['var'].render_jcb(self.task_config, 'soca_obs_staging')
+        FileHandler(obs_list).sync()
+
         # stage the soca utility yamls (gridgen, fields and ufo mapping yamls)
         logger.info(f"Staging SOCA utility yaml files from {self.task_config.PARMsoca}")
         soca_utility_list = parse_j2yaml(self.task_config.MARINE_UTILITY_YAML_TMPL, self.task_config)
         FileHandler(soca_utility_list).sync()
 
-    @logit(logger)
-    def _prep_variational_yaml(self: Task) -> None:
-        """Create the yaml configuration to run the SOCA variational application
-        """
+        # stage the ocean and ice backgrounds for FGAT
+        logger.info(f"Staging files needed for deterministic analysis from COM")
+        bkg_list = parse_j2yaml(self.task_config.MARINE_DET_STAGE_BKG_YAML_TMPL, self.task_config)
+        FileHandler(bkg_list).sync()
 
-        # prepare background list for the pseudo model, check bkg date for consistency
-        mdau.gen_bkg_list(bkg_path='./bkg',
-                          window_begin=self.task_config.MARINE_WINDOW_BEGIN,
-                          yaml_name='bkg_list.yaml')
+        # stage files and link directories from B-matrix job needed for deterministic analysis
+        logger.info(f"Staging files needed for deterministic analysis from COM")
+        soca_files_list = parse_j2yaml(self.task_config.MARINE_DET_STAGE_FILES_YAML_TMPL, self.task_config)
+        FileHandler(soca_files_list).sync()
 
-        # Make a copy of the env config before modifying to avoid breaking something else
-        envconfig_jcb = copy.deepcopy(self.task_config)
-        logger.info(f"---------------- Prepare the yaml configuration")
-        logger.info(f"{envconfig_jcb}")       # Prepare the yaml configuration
+        # assert that dates of the history files are correct
+        mdau.test_hist_date('./INPUT/MOM.res.nc', self.task_config.MARINE_WINDOW_BEGIN)
+        for state in self.task_config.marine_pseudo_model_states:
+            mdau.test_hist_date(state['basename'] + state['ocn_filename'],
+                                datetime.strptime(state['date'], '%Y-%m-%dT%H:%M:%SZ'))
 
-        # Add the things to the envconfig in order to template JCB files
-        envconfig_jcb['PARMgfs'] = self.task_config.PARMgfs
-        envconfig_jcb['NMEM_ENS'] = self.task_config.NMEM_ENS
-        envconfig_jcb['berror_model'] = 'marine_background_error_static_diffusion'
-        if self.task_config.NMEM_ENS >= 2:
-            envconfig_jcb['berror_model'] = 'marine_background_error_hybrid_diffusion_diffusion'
-        envconfig_jcb['DATA'] = self.task_config.DATA
-        envconfig_jcb['OPREFIX'] = self.task_config.OPREFIX
-        envconfig_jcb['PDY'] = os.getenv('PDY')
-        envconfig_jcb['cyc'] = os.getenv('cyc')
-        envconfig_jcb['SOCA_NINNER'] = self.task_config.SOCA_NINNER
-        envconfig_jcb['HOMEgfs'] = self.task_config.HOMEgfs
-        envconfig_jcb['DO_TEST_MODE'] = self.task_config.DO_TEST_MODE
-        envconfig_jcb['RUN'] = self.task_config.RUN
-        envconfig_jcb['current_cycle'] = self.task_config.current_cycle
-        envconfig_jcb['MOM6_LEVS'] = mdau.get_mom6_levels(str(self.task_config.OCNRES).zfill(3))
-        envconfig_jcb['observations'] = self.task_config.observations
-
-        # Write obs_list_short
-        save_as_yaml(self.task_config['obs_list'], 'obs_list_short.yaml')
-        os.environ['OBS_LIST_SHORT'] = 'obs_list_short.yaml'
-
-        # Render the JCB configuration files
-        jcb_base_yaml = os.path.join(self.task_config.PARMsoca, 'marine-jcb-base.yaml')
-        jcb_algo_yaml = self.task_config.JCB_ALGO_YAML_VAR
-
-        jcb_base_config = parse_j2yaml(path=jcb_base_yaml, data=envconfig_jcb)
-        jcb_algo_config = parse_j2yaml(path=jcb_algo_yaml, data=envconfig_jcb)
-
-        # Override base with the application specific config
-        jcb_config = {**jcb_base_config, **jcb_algo_config}
-
-        # convert datetime to string
-        jcb_config['window_begin'] = self.task_config.MARINE_WINDOW_BEGIN.strftime('%Y-%m-%dT%H:%M:%SZ')
-        jcb_config['window_middle'] = self.task_config.MARINE_WINDOW_MIDDLE.strftime('%Y-%m-%dT%H:%M:%SZ')
-
-        # Current hack so that this is not done directly in the JCB base yaml
-        jcb_config['marine_pseudo_model_states'] = parse_yaml('bkg_list.yaml')
-
-        # Render the full JEDI configuration file using JCB
-        jedi_config = render(jcb_config)
-
-        # Save the JEDI configuration file
-        var_yaml_jcb = 'var.yaml'
-        mdau.clean_empty_obsspaces(jedi_config, target=var_yaml_jcb, app='var')
-
-    def _prep_checkpoint(self: Task) -> None:
-        """Create the yaml configuration to run the SOCA to MOM6 IAU increment
-        """
-        # prepare the socaincr2mom6.yaml
-        logger.info("Generate the SOCA to MOM6 IAU increment YAML file")
-        data = {'marine_window_begin': self.task_config.MARINE_WINDOW_BEGIN_ISO,
-                'marine_window_middle': self.task_config.MARINE_WINDOW_MIDDLE_ISO}
-        soca2mom6inc_config = parse_j2yaml(path=os.path.join(self.task_config.MARINE_JCB_GDAS_ALGO, 'socaincr2mom6.yaml.j2'),
-                                           data=data)
-        soca2mom6inc_config.save(os.path.join(self.task_config.DATA, 'socaincr2mom6.yaml'))
-
-        # prepare the SOCA to CICE YAML file
-        logger.info("Generate the SOCA to CICE RST YAML file")
-
-        # set the restart date, dependent on the cycling type
-        if self.task_config.DOIAU:
-            # forecast initialized at the begining of the DA window
-            fcst_begin = self.task_config.MARINE_WINDOW_BEGIN_ISO
-            rst_date = self.task_config.MARINE_WINDOW_BEGIN.strftime('%Y%m%d.%H%M%S')
-        else:
-            # forecast initialized at the middle of the DA window
-            fcst_begin = self.task_config.MARINE_WINDOW_MIDDLE_ISO
-            rst_date = self.task_config.MARINE_WINDOW_MIDDLE.strftime('%Y%m%d.%H%M%S')
-
-        # make a copy of the CICE6 restart
-        ice_rst = os.path.join(self.task_config.COMIN_ICE_RESTART_PREV, f'{rst_date}.cice_model.res.nc')
-        ice_rst_ana = os.path.join(self.task_config.DATA, 'Data', rst_date + '.cice_model.res.nc')
-        FileHandler({'copy': [[ice_rst, ice_rst_ana]]}).sync()
-
-        # prepare the necessary configuration for the SOCA to CICE application
-        soca2cice_param = AttrDict({
-            "ocn_ana": f"./Data/ocn.3dvarfgat_pseudo.an.{self.task_config.MARINE_WINDOW_MIDDLE_ISO}.nc",
-            "ice_ana": f"./Data/ice.3dvarfgat_pseudo.an.{self.task_config.MARINE_WINDOW_MIDDLE_ISO}.nc",
-            "ocn_inc": f"./Data/ocn.3dvarfgat_pseudo.incr.{self.task_config.MARINE_WINDOW_MIDDLE_ISO}.nc",
-            "ice_inc": f"./Data/ice.3dvarfgat_pseudo.incr.{self.task_config.MARINE_WINDOW_MIDDLE_ISO}.nc",
-            "ice_rst": ice_rst_ana,
-            "fcst_begin": fcst_begin
-        })
-        logger.debug(f"{soca2cice_param}")
-
-        # render the SOCA to CICE YAML file for the Arctic and Antarctic
-        logger.info("render the SOCA to CICE YAML file for the Arctic and Antarctic")
-        varchgyaml = 'soca_2cice_global.yaml'
-        soca2cice_config = parse_j2yaml(path=os.path.join(self.task_config.MARINE_JCB_GDAS_ALGO, f'{varchgyaml}.j2'),
-                                        data=soca2cice_param)
-        soca2cice_config.save(os.path.join(self.task_config.DATA, varchgyaml))
+        # initialize JEDI variational application
+        self.jedi_dict['var'].initialize(self.task_config)
+        self.jedi_dict['socaincr2mom6'].initialize(self.task_config)
+        self.jedi_dict['soca_2cice_global'].initialize(self.task_config)
+        self.jedi_dict['soca_diag_stats'].initialize(self.task_config)
path: ush/python/pygfs/task/marine_analysis.py
user: login: aerorahul
body: One suggestion to insert `logger.info` between each of these initialize calls to make it easy to trace the execution.
created_at: 2025-05-14T13:42:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,452 +48,158 @@ def __init__(self, config):
         else:
             _enspert_relpath = None
 
+        # Determine background error model
+        if self.task_config.NMEM_ENS >= 2:
+            _berror_model = 'marine_background_error_hybrid_diffusion_diffusion'
+        else:
+            _berror_model = 'marine_background_error_static_diffusion'
+
         # Create a local dictionary that is repeatedly used across this class
         local_dict = AttrDict(
             {
                 'PARMsoca': os.path.join(self.task_config.PARMgfs, 'gdas', 'soca'),
                 'MARINE_WINDOW_BEGIN': _window_begin,
-                'MARINE_WINDOW_BEGIN_ISO': _window_begin.strftime('%Y-%m-%dT%H:%M:%SZ'),
                 'MARINE_WINDOW_END': _window_end,
-                'MARINE_WINDOW_LENGTH': f"PT{self.task_config['assim_freq']}H",
                 'MARINE_WINDOW_MIDDLE': self.task_config.current_cycle,
-                'MARINE_WINDOW_MIDDLE_ISO': self.task_config.current_cycle.strftime('%Y-%m-%dT%H:%M:%SZ'),
+                'MARINE_WINDOW_LENGTH': f"PT{self.task_config['assim_freq']}H",
+                'MARINE_WINDOW_BEGIN_ISO': to_isotime(_window_begin),
+                'MARINE_WINDOW_MIDDLE_ISO': to_isotime(self.task_config.current_cycle),
                 'ENSPERT_RELPATH': _enspert_relpath,
                 'CALC_SCALE_EXEC': _calc_scale_exec,
                 'OPREFIX': f"{self.task_config.RUN}.t{self.task_config.cyc:02d}z.",
                 'APREFIX': f"{self.task_config.RUN}.t{self.task_config.cyc:02d}z.",
-                'app_path_observations': self.task_config.MARINE_JCB_GDAS_OBS
+                'berror_model': _berror_model,
+                'MOM6_LEVS': mdau.get_mom6_levels(str(self.task_config.OCNRES).zfill(3)),
+                'app_path_observations': self.task_config.MARINE_JCB_GDAS_OBS,
+                'marine_pseudo_model_states': mdau.gen_bkg_list(bkg_path='./bkg',
+                                                                window_begin=_window_begin)
             }
         )
 
         # Extend task_config with local_dict
         self.task_config.update(local_dict)
 
+        # Construct dictionary of JEDI objects, one for each JEDI application need for the analysis
+        expected_keys = ['var', 'socaincr2mom6', 'soca_2cice_global', 'soca_diag_stats']
+        self.jedi_dict = Jedi.get_jedi_dict(self.task_config.JEDI_CONFIG_YAML_ANALYSIS, self.task_config, expected_keys)
+
     @logit(logger)
     def initialize(self: Task) -> None:
-        """Initialize the marine analysis
+        """Initialize the marine analysis task
 
         This method will initialize the marine analysis.
         This includes:
-        - staging the deterministic backgrounds (middle of window)
         - staging SOCA fix files
-        - staging static ensemble members (optional)
-        - staging ensemble members (optional)
-        - generating the YAML files for the JEDI and GDASApp executables
-        - creating output directories
-        """
-        super().initialize()
-
-        # prepare the directory structure to run SOCA
-        self._prep_scratch_dir()
-
-        # fetch observations from COMROOT
-        # TODO(G.V. or A.E.): Keep a copy of the obs in the scratch fs after the obs prep job
-        self._fetch_observations()
-
-        # stage the ocean and ice backgrounds for FGAT
-        bkg_list = parse_j2yaml(self.task_config.MARINE_DET_STAGE_BKG_YAML_TMPL, self.task_config)
-        FileHandler(bkg_list).sync()
-
-        # stage the soca grid
-        FileHandler({'copy': [[os.path.join(self.task_config.COMIN_OCEAN_BMATRIX, 'soca_gridspec.nc'),
-                               os.path.join(self.task_config.DATA, 'soca_gridspec.nc')]]}).sync()
-
-        # link the flow dependent static B resources from the B-matrix task of the same cycle
-        os.symlink('../staticb', 'staticb')
-
-        # hybrid EnVAR case
-        if self.task_config.DOHYBVAR_OCN == "YES" or self.task_config.NMEM_ENS >= 2:
-            # stage the ensemble weights
-            logger.debug(f"Stage ensemble weights for the hybrid background error")
-            FileHandler({'copy': [[os.path.join(self.task_config.COMIN_OCEAN_BMATRIX, f'{self.task_config.APREFIX}ocean.ens_weights.nc'),
-                                   os.path.join(self.task_config.DATA, 'ocean.ens_weights.nc')],
-                                  [os.path.join(self.task_config.COMIN_ICE_BMATRIX, f'{self.task_config.APREFIX}ice.ens_weights.nc'),
-                                   os.path.join(self.task_config.DATA, 'ice.ens_weights.nc')]]}).sync()
-
-        # prepare the yaml configuration to run the SOCA variational application
-        self._prep_variational_yaml()
-
-        # prepare the yaml configuration to run the SOCA to MOM6 IAU increment
-        self._prep_checkpoint()
-
-    @logit(logger)
-    def _fetch_observations(self: Task) -> None:
-        """Fetch observations from COMIN_OBS
-
-        This method will fetch the observations for the cycle and check the
-        list against what is available for the cycle.
-        """
-
-        # get the list of observations
-
-        # "observations" is expected by later JCB code to populate it with config info,
-        # but the obs_list as such is needed later
-        self.task_config.observations = parse_j2yaml(self.task_config.MARINE_OBS_LIST_YAML, self.task_config)['observations']
-        self.task_config.obs_list = self.task_config.observations
-
-        obsconfigfile = os.path.join(self.task_config['PARMgfs'], 'gdas/soca/obs/obs_list_base_yaml.j2')
-        self.task_config.observations = parse_j2yaml(obsconfigfile, self.task_config)
-
-        obs_files = []
-
-        for observer in self.task_config['observations']['observers']:
-            filename = f"{self.task_config.OPREFIX}{observer['obs space']['name'].lower()}.{to_YMD(self.task_config.PDY)}{self.task_config.cyc:02d}.nc4"
-            logger.info(f"******** {filename}")
-            obs_files.append(filename)
-
-        obs_files_to_copy = []
-
-        # copy obs from COMIN_OBS to DATA/obs
-        for obs_file in obs_files:
-            logger.info(f"******* {obs_file}")
-            obs_src = os.path.join(self.task_config.COMIN_OBS, obs_file)
-            obs_dst = os.path.join(self.task_config.DATA, 'obs', obs_file)
-            logger.info(f"******* {obs_src}")
-            if os.path.exists(obs_src):
-                logger.info(f"******* fetching {obs_file}")
-                obs_files_to_copy.append([obs_src, obs_dst])
-            else:
-                logger.info(f"******* {obs_file} is not in the database")
+        - preparing the namelists for deterministic MOM6 and analysis geometry
+        - staging observations
+        - staging input YAMLs for SOCA utilities
+        - staging the deterministic backgrounds (middle of window)
+        - staging files and link directories from B-matrix job needed for deterministic analysis
+        - generating list of model pseudo states
+        - initializing all the JEDI applications required for the marine analysis
 
-        FileHandler({'copy': obs_files_to_copy}).sync()
+        Parameters
+        ----------
+        None
 
-    @logit(logger)
-    def _prep_scratch_dir(self: Task) -> None:
-        """Create and stage all the resources needed to run SOCA/JEDI, including the necesssary
-           directory structure to run the SOCA variational application
+        Returns
+        ----------
+        None
         """
-        logger.info(f"---------------- Setup runtime environement")
-
-        anl_dir = self.task_config.DATA
-
-        # create analysis directories
-        diags = os.path.join(anl_dir, 'diags')            # output dir for soca DA obs space
-        obs_in = os.path.join(anl_dir, 'obs')             # input      "           "
-        anl_out = os.path.join(anl_dir, 'Data')           # output dir for soca DA
-        FileHandler({'mkdir': [diags, obs_in, anl_out]}).sync()
 
         # stage fix files
         logger.info(f"Staging SOCA fix files from {self.task_config.SOCA_INPUT_FIX_DIR}")
         soca_fix_list = parse_j2yaml(self.task_config.SOCA_FIX_YAML_TMPL, self.task_config)
         FileHandler(soca_fix_list).sync()
 
         # prepare the deterministic MOM6 input.nml
+        logger.info(f"Preparing deterministic MOM6 input namelist")
         mdau.prep_input_nml(self.task_config)
 
         # prepare the input.nml for the analysis geometry
+        logger.info(f"Preparing analysis geometry input namelist")
         mdau.prep_input_nml(self.task_config, output_nml="./anl_geom/mom_input.nml",
                             simple_geom=True, mom_input="./anl_geom/MOM_input")
 
+        # fetch observations from COMROOT
+        # TODO(G.V. or A.E.): Keep a copy of the obs in the scratch fs after the obs prep job
+        logger.info(f"Staging observations from {self.task_config.COMIN_OBS}")
+        obs_list = self.jedi_dict['var'].render_jcb(self.task_config, 'soca_obs_staging')
+        FileHandler(obs_list).sync()
+
         # stage the soca utility yamls (gridgen, fields and ufo mapping yamls)
         logger.info(f"Staging SOCA utility yaml files from {self.task_config.PARMsoca}")
         soca_utility_list = parse_j2yaml(self.task_config.MARINE_UTILITY_YAML_TMPL, self.task_config)
         FileHandler(soca_utility_list).sync()
 
-    @logit(logger)
-    def _prep_variational_yaml(self: Task) -> None:
-        """Create the yaml configuration to run the SOCA variational application
-        """
-
-        # prepare background list for the pseudo model, check bkg date for consistency
-        mdau.gen_bkg_list(bkg_path='./bkg',
-                          window_begin=self.task_config.MARINE_WINDOW_BEGIN,
-                          yaml_name='bkg_list.yaml')
+        # stage the ocean and ice backgrounds for FGAT
+        logger.info(f"Staging files needed for deterministic analysis from COM")
+        bkg_list = parse_j2yaml(self.task_config.MARINE_DET_STAGE_BKG_YAML_TMPL, self.task_config)
+        FileHandler(bkg_list).sync()
 
-        # Make a copy of the env config before modifying to avoid breaking something else
-        envconfig_jcb = copy.deepcopy(self.task_config)
-        logger.info(f"---------------- Prepare the yaml configuration")
-        logger.info(f"{envconfig_jcb}")       # Prepare the yaml configuration
+        # stage files and link directories from B-matrix job needed for deterministic analysis
+        logger.info(f"Staging files needed for deterministic analysis from COM")
+        soca_files_list = parse_j2yaml(self.task_config.MARINE_DET_STAGE_FILES_YAML_TMPL, self.task_config)
+        FileHandler(soca_files_list).sync()
 
-        # Add the things to the envconfig in order to template JCB files
-        envconfig_jcb['PARMgfs'] = self.task_config.PARMgfs
-        envconfig_jcb['NMEM_ENS'] = self.task_config.NMEM_ENS
-        envconfig_jcb['berror_model'] = 'marine_background_error_static_diffusion'
-        if self.task_config.NMEM_ENS >= 2:
-            envconfig_jcb['berror_model'] = 'marine_background_error_hybrid_diffusion_diffusion'
-        envconfig_jcb['DATA'] = self.task_config.DATA
-        envconfig_jcb['OPREFIX'] = self.task_config.OPREFIX
-        envconfig_jcb['PDY'] = os.getenv('PDY')
-        envconfig_jcb['cyc'] = os.getenv('cyc')
-        envconfig_jcb['SOCA_NINNER'] = self.task_config.SOCA_NINNER
-        envconfig_jcb['HOMEgfs'] = self.task_config.HOMEgfs
-        envconfig_jcb['DO_TEST_MODE'] = self.task_config.DO_TEST_MODE
-        envconfig_jcb['RUN'] = self.task_config.RUN
-        envconfig_jcb['current_cycle'] = self.task_config.current_cycle
-        envconfig_jcb['MOM6_LEVS'] = mdau.get_mom6_levels(str(self.task_config.OCNRES).zfill(3))
-        envconfig_jcb['observations'] = self.task_config.observations
-
-        # Write obs_list_short
-        save_as_yaml(self.task_config['obs_list'], 'obs_list_short.yaml')
-        os.environ['OBS_LIST_SHORT'] = 'obs_list_short.yaml'
-
-        # Render the JCB configuration files
-        jcb_base_yaml = os.path.join(self.task_config.PARMsoca, 'marine-jcb-base.yaml')
-        jcb_algo_yaml = self.task_config.JCB_ALGO_YAML_VAR
-
-        jcb_base_config = parse_j2yaml(path=jcb_base_yaml, data=envconfig_jcb)
-        jcb_algo_config = parse_j2yaml(path=jcb_algo_yaml, data=envconfig_jcb)
-
-        # Override base with the application specific config
-        jcb_config = {**jcb_base_config, **jcb_algo_config}
-
-        # convert datetime to string
-        jcb_config['window_begin'] = self.task_config.MARINE_WINDOW_BEGIN.strftime('%Y-%m-%dT%H:%M:%SZ')
-        jcb_config['window_middle'] = self.task_config.MARINE_WINDOW_MIDDLE.strftime('%Y-%m-%dT%H:%M:%SZ')
-
-        # Current hack so that this is not done directly in the JCB base yaml
-        jcb_config['marine_pseudo_model_states'] = parse_yaml('bkg_list.yaml')
-
-        # Render the full JEDI configuration file using JCB
-        jedi_config = render(jcb_config)
-
-        # Save the JEDI configuration file
-        var_yaml_jcb = 'var.yaml'
-        mdau.clean_empty_obsspaces(jedi_config, target=var_yaml_jcb, app='var')
-
-    def _prep_checkpoint(self: Task) -> None:
-        """Create the yaml configuration to run the SOCA to MOM6 IAU increment
-        """
-        # prepare the socaincr2mom6.yaml
-        logger.info("Generate the SOCA to MOM6 IAU increment YAML file")
-        data = {'marine_window_begin': self.task_config.MARINE_WINDOW_BEGIN_ISO,
-                'marine_window_middle': self.task_config.MARINE_WINDOW_MIDDLE_ISO}
-        soca2mom6inc_config = parse_j2yaml(path=os.path.join(self.task_config.MARINE_JCB_GDAS_ALGO, 'socaincr2mom6.yaml.j2'),
-                                           data=data)
-        soca2mom6inc_config.save(os.path.join(self.task_config.DATA, 'socaincr2mom6.yaml'))
-
-        # prepare the SOCA to CICE YAML file
-        logger.info("Generate the SOCA to CICE RST YAML file")
-
-        # set the restart date, dependent on the cycling type
-        if self.task_config.DOIAU:
-            # forecast initialized at the begining of the DA window
-            fcst_begin = self.task_config.MARINE_WINDOW_BEGIN_ISO
-            rst_date = self.task_config.MARINE_WINDOW_BEGIN.strftime('%Y%m%d.%H%M%S')
-        else:
-            # forecast initialized at the middle of the DA window
-            fcst_begin = self.task_config.MARINE_WINDOW_MIDDLE_ISO
-            rst_date = self.task_config.MARINE_WINDOW_MIDDLE.strftime('%Y%m%d.%H%M%S')
-
-        # make a copy of the CICE6 restart
-        ice_rst = os.path.join(self.task_config.COMIN_ICE_RESTART_PREV, f'{rst_date}.cice_model.res.nc')
-        ice_rst_ana = os.path.join(self.task_config.DATA, 'Data', rst_date + '.cice_model.res.nc')
-        FileHandler({'copy': [[ice_rst, ice_rst_ana]]}).sync()
-
-        # prepare the necessary configuration for the SOCA to CICE application
-        soca2cice_param = AttrDict({
-            "ocn_ana": f"./Data/ocn.3dvarfgat_pseudo.an.{self.task_config.MARINE_WINDOW_MIDDLE_ISO}.nc",
-            "ice_ana": f"./Data/ice.3dvarfgat_pseudo.an.{self.task_config.MARINE_WINDOW_MIDDLE_ISO}.nc",
-            "ocn_inc": f"./Data/ocn.3dvarfgat_pseudo.incr.{self.task_config.MARINE_WINDOW_MIDDLE_ISO}.nc",
-            "ice_inc": f"./Data/ice.3dvarfgat_pseudo.incr.{self.task_config.MARINE_WINDOW_MIDDLE_ISO}.nc",
-            "ice_rst": ice_rst_ana,
-            "fcst_begin": fcst_begin
-        })
-        logger.debug(f"{soca2cice_param}")
-
-        # render the SOCA to CICE YAML file for the Arctic and Antarctic
-        logger.info("render the SOCA to CICE YAML file for the Arctic and Antarctic")
-        varchgyaml = 'soca_2cice_global.yaml'
-        soca2cice_config = parse_j2yaml(path=os.path.join(self.task_config.MARINE_JCB_GDAS_ALGO, f'{varchgyaml}.j2'),
-                                        data=soca2cice_param)
-        soca2cice_config.save(os.path.join(self.task_config.DATA, varchgyaml))
+        # assert that dates of the history files are correct
+        mdau.test_hist_date('./INPUT/MOM.res.nc', self.task_config.MARINE_WINDOW_BEGIN)
+        for state in self.task_config.marine_pseudo_model_states:
+            mdau.test_hist_date(state['basename'] + state['ocn_filename'],
+                                datetime.strptime(state['date'], '%Y-%m-%dT%H:%M:%SZ'))
 
-    @logit(logger)
-    def variational(self: Task) -> None:
-        # link gdas_soca_gridgen.x
-        mdau.link_executable(self.task_config, 'gdas.x')
-        exec_cmd = Executable(self.task_config.APRUN_MARINEANLVAR)
-        exec_name = os.path.join(self.task_config.DATA, 'gdas.x')
-        exec_cmd.add_default_arg(exec_name)
-        exec_cmd.add_default_arg('soca')
-        exec_cmd.add_default_arg('variational')
-        exec_cmd.add_default_arg('var.yaml')
-
-        mdau.run(exec_cmd)
+        # initialize JEDI applications
+        logger.info(f"Initializing SOCA variational application")
+        self.jedi_dict['var'].initialize(self.task_config)
 
-    @logit(logger)
-    def checkpoint_cice6(self: Task, soca2ciceyaml) -> None:
-        # link gdas_soca_gridgen.x
-        mdau.link_executable(self.task_config, 'gdas.x')
-        exec_cmd = Executable(self.task_config.APRUN_MARINEANLCHKPT)
-        exec_name = os.path.join(self.task_config.DATA, 'gdas.x')
-        exec_cmd.add_default_arg(exec_name)
-        exec_cmd.add_default_arg('soca')
-        exec_cmd.add_default_arg('convertstate')
-        exec_cmd.add_default_arg(soca2ciceyaml)
-
-        mdau.run(exec_cmd)
+        logger.info(f"Initializing SOCA increment handler")
+        self.jedi_dict['socaincr2mom6'].initialize(self.task_config)
 
-    @logit(logger)
-    def checkpoint_mom6_iau(self: Task, socaincr2mom6yaml) -> None:
-        # link gdas_incr_handler.x
-        mdau.link_executable(self.task_config, 'gdas_incr_handler.x')
-        exec_cmd = Executable(self.task_config.APRUN_MARINEANLCHKPT)
-        exec_name = os.path.join(self.task_config.DATA, 'gdas_incr_handler.x')
-        exec_cmd.add_default_arg(exec_name)
-        exec_cmd.add_default_arg(socaincr2mom6yaml)
+        logger.info(f"Initializing SOCA convertstate application")
+        self.jedi_dict['soca_2cice_global'].initialize(self.task_config)
 
-        mdau.run(exec_cmd)
+        logger.info(f"Initializing SOCA observation statistics application")
+        self.jedi_dict['soca_diag_stats'].initialize(self.task_config)
 
     @logit(logger)
-    def finalize(self: Task) -> None:
-        """Finalize the marine analysis job
-           This method saves the results of the deterministic variational analysis to the COMROOT
-        """
+    def execute(self, jedi_dict_key: str) -> None:
+        """Execute JEDI application of marine analysis
 
-        def list_all_files(dir_in, dir_out, wc='*', fh_list=[]):
-            files = glob.glob(os.path.join(dir_in, wc))
-            for file_src in files:
-                file_dst = os.path.join(dir_out, os.path.basename(file_src))
-                fh_list.append([file_src, file_dst])
-            return fh_list
-
-        # variables of convenience
-        com_ocean_analysis = self.task_config.COMOUT_OCEAN_ANALYSIS
-        com_ice_analysis = self.task_config.COMOUT_ICE_ANALYSIS
-        com_ice_restart = self.task_config.COMOUT_ICE_RESTART
-        anl_dir = self.task_config.DATA
-        cdate = self.task_config.CDATE
-        pdy = self.task_config.PDY
-        staticsoca_dir = self.task_config.SOCA_INPUT_FIX_DIR
-        RUN = self.task_config.RUN
-        cyc = str(self.task_config.cyc).zfill(2)
-        bcyc = str(self.task_config.MARINE_WINDOW_BEGIN.hour).zfill(2)
-        bdate = self.task_config.MARINE_WINDOW_BEGIN_ISO
-        mdate = self.task_config.MARINE_WINDOW_MIDDLE_ISO
-        nmem_ens = int(self.task_config.NMEM_ENS)
-
-        logger.info(f"---------------- Copy from RUNDIR to COMOUT")
-
-        post_file_list = []
-
-        # Make a copy the IAU increment
-        post_file_list.append([os.path.join(anl_dir, 'ocn.inc.nc'),
-                               os.path.join(com_ocean_analysis, f'{RUN}.t{cyc}z.ocninc.nc')])
-
-        domains = ['ocn', 'ice']
-        for domain in domains:
-            '''
-            # Copy of the diagonal of the background error for the cycle
-            post_file_list.append([os.path.join(anl_dir, f'{domain}.bkgerr_stddev.incr.{mdate}.nc'),
-                                   os.path.join(com_ocean_analysis, f'{RUN}.t{cyc}z.{domain}.bkgerr_stddev.nc')])
-
-            # Copy the recentering error
-            if nmem_ens > 2:
-                post_file_list.append([os.path.join(anl_dir, 'static_ens', f'{domain}.ssh_recentering_error.incr.{bdate}.nc'),
-                                       os.path.join(com_ocean_analysis, f'{RUN}.t{cyc}z.{domain}.recentering_error.nc')])
-            '''
-
-            # Copy the ice and ocean increments
-            post_file_list.append([os.path.join(anl_dir, 'Data', f'{domain}.3dvarfgat_pseudo.incr.{mdate}.nc'),
-                                   os.path.join(com_ocean_analysis, f'{RUN}.t{cyc}z.{domain}.incr.nc')])
-
-            # Copy the analysis at the start of the window
-            post_file_list.append([os.path.join(anl_dir, 'Data', f'{domain}.3dvarfgat_pseudo.an.{mdate}.nc'),
-                                   os.path.join(com_ocean_analysis, f'{RUN}.t{cyc}z.{domain}ana.nc')])
path: ush/python/pygfs/task/marine_analysis.py
user: login: shlyaeva
body: Here's the reason for the failure: compare with https://github.com/NOAA-EMC/GDASApp/blob/46f0ef62fcd4514a8b9feeffab1088f390c72283/parm/soca/soca_det_finalize.yaml.j2#L20 that added a dot.
created_at: 2025-05-20T15:15:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -257,6 +261,39 @@ def get_jedi_dict(jedi_config_yaml: str, task_config: AttrDict, expected_block_n
         # Return dictionary of JEDI objects
         return jedi_dict
 
+    @logit(logger)
+    def clean_empty_obsspaces(self):
path: ush/python/pygfs/jedi/jedi.py
user: login: CoryMartin-NOAA
body: This method makes me nervous. Are we 100% sure this doesn't break anything?
created_at: 2025-05-21T14:44:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -257,6 +261,39 @@ def get_jedi_dict(jedi_config_yaml: str, task_config: AttrDict, expected_block_n
         # Return dictionary of JEDI objects
         return jedi_dict
 
+    @logit(logger)
+    def clean_empty_obsspaces(self):
path: ush/python/pygfs/jedi/jedi.py
user: login: DavidNew-NOAA
body: It can be tricky, such as the case of the LETKF solver, where you need to initialize the YAML _after_ the LETKF observer runs, because otherwise all the obs spaces will be removed since the diag files don't yet exist. But right now it doesn't break anything.
created_at: 2025-05-21T14:50:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -257,6 +261,39 @@ def get_jedi_dict(jedi_config_yaml: str, task_config: AttrDict, expected_block_n
         # Return dictionary of JEDI objects
         return jedi_dict
 
+    @logit(logger)
+    def clean_empty_obsspaces(self):
path: ush/python/pygfs/jedi/jedi.py
user: login: CoryMartin-NOAA
body: Ok, I just saw this https://github.com/NOAA-EMC/global-workflow/pull/3645#issuecomment-2898164952 and thought it might be related
created_at: 2025-05-21T14:51:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -143,6 +140,10 @@ def initialize(self) -> None:
         ]
         FileHandler({'mkdir': newdirs}).sync()
 
+        # initialize JEDI variational application
+        logger.info(f"Initializing JEDI variational DA application")
+        self.jedi_dict['snowanlvar'].initialize(self.task_config, clean_empty_obsspaces=True)
path: ush/python/pygfs/task/snow_analysis.py
user: login: CoryMartin-NOAA
body: is this line the reason why the snow DA fails? Can we try it without cleaning the obsspace?
created_at: 2025-05-22T12:37:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -257,6 +261,39 @@ def get_jedi_dict(jedi_config_yaml: str, task_config: AttrDict, expected_block_n
         # Return dictionary of JEDI objects
         return jedi_dict
 
+    @logit(logger)
+    def clean_empty_obsspaces(self):
path: ush/python/pygfs/jedi/jedi.py
user: login: aerorahul
body: @CoryMartin-NOAA 
I am about to merge this PR since the test of Gaea passed overnight.  I was just going over comments, is this ok to be resolved?
created_at: 2025-05-23T12:45:33Z
author_association: CONTRIBUTOR

PR comments:
user: login: DavidNew-NOAA
created_at: 2025-05-02T20:03:15Z
body: @guillaumevernieres and @AndrewEichmann-NOAA Maybe we can tag up on Monday, and I can go over these changes before you review them since they're pretty extensive. No features have been changed, but things have been moved around quite a bit.
author_association: CONTRIBUTOR
user: login: guillaumevernieres
created_at: 2025-05-05T12:56:25Z
body: > @guillaumevernieres and @AndrewEichmann-NOAA Maybe we can tag up on Monday, and I can go over these changes before you review them since they're pretty extensive. No features have been changed, but things have been moved around quite a bit.

Sounds good @DavidNew-NOAA , can we tag-up Thursday instead? Pinging @shlyaeva too ... 
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-05T12:57:46Z
body: @guillaumevernieres Sure, I'll be there in person on Thursday
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-15T18:03:38Z
body: @KateFriedman-NOAA Can you stop CI. The companion GDASApp PR wasn't merged. I just merged it and will update the gdas hash momentarily
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-15T18:06:01Z
body: OK, GDASApp hash updated. _Now_ ready for CI.
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-15T18:09:01Z
body: > @KateFriedman-NOAA Can you stop CI. The companion GDASApp PR wasn't merged. I just merged it and will update the gdas hash momentarily

Or @TerrenceMcGuinness-NOAA 
author_association: CONTRIBUTOR
user: login: emcbot
created_at: 2025-05-15T19:12:04Z
body: Experiment C48mx500_hybAOWCDA **FAILED** on Hera in Build# 2 with error logs:
```
/scratch1/NCEPDEV/global/glopara/CI/3642/RUNTESTS/COMROOT/C48mx500_hybAOWCDA_e08c9b84/logs/2021032500/gdas_marineanlletkf.log
```

Follow link here to view the contents of the above file(s): [(gdas_marineanlletkf.log)](https://gist.github.com/emcbot/2f05dadb16ad8e07063ca8904a3e288b)
author_association: NONE
user: login: emcbot
created_at: 2025-05-15T19:12:14Z
body: Experiment C48mx500_hybAOWCDA **FAILED** on Hera in Build# 2 in
`/scratch1/NCEPDEV/global/glopara/CI/3642/RUNTESTS/EXPDIR/C48mx500_hybAOWCDA_e08c9b84`
author_association: NONE
user: login: emcbot
created_at: 2025-05-15T19:13:19Z
body: Experiment C48mx500_hybAOWCDA **FAILED** on Gaeac6 in Build# 1 with error logs:
```
/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/COMROOT/C48mx500_hybAOWCDA_e08c9b84/logs/2021032500/gdas_marineanlletkf.log
```

Follow link here to view the contents of the above file(s): [(gdas_marineanlletkf.log)](https://gist.github.com/emcbot/6227df41950426ef3dffe6149276d418)
author_association: NONE
user: login: emcbot
created_at: 2025-05-15T19:13:27Z
body: Experiment C48mx500_hybAOWCDA **FAILED** on Gaeac6 in Build# 1 in
`/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/EXPDIR/C48mx500_hybAOWCDA_e08c9b84`
author_association: NONE
user: login: emcbot
created_at: 2025-05-15T19:19:22Z
body: Experiment C48mx500_3DVarAOWCDA **FAILED** on Gaeac6 in Build# 1 with error logs:
```
/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_e08c9b84/logs/2021032500/gdas_marineanlchkpt.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_e08c9b84/logs/2021032500/gfs_marineanlchkpt.log
```

Follow link here to view the contents of the above file(s): [(gdas_marineanlchkpt.log)](https://gist.github.com/emcbot/3ce7347c4302d9ffe0008a9d9822bd3a) [(gfs_marineanlchkpt.log)](https://gist.github.com/emcbot/3c1329ba4c43d6de0245a82052913c49)
author_association: NONE
user: login: emcbot
created_at: 2025-05-15T19:19:32Z
body: Experiment C48mx500_3DVarAOWCDA **FAILED** on Gaeac6 in Build# 1 in
`/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_e08c9b84`
author_association: NONE
user: login: emcbot
created_at: 2025-05-15T19:19:38Z
body: Experiment C96C48mx500_S2SW_cyc_gfs **FAILED** on Gaeac6 in Build# 1 with error logs:
```
/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_e08c9b84/logs/2021122018/enkfgdas_esnowanl.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_e08c9b84/logs/2021122018/enkfgfs_esnowanl.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_e08c9b84/logs/2021122018/gdas_snowanl.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_e08c9b84/logs/2021122018/gfs_snowanl.log
```

Follow link here to view the contents of the above file(s): [(enkfgdas_esnowanl.log)](https://gist.github.com/emcbot/abf405a32bf436119fb093a8602dd4e9) [(enkfgfs_esnowanl.log)](https://gist.github.com/emcbot/ec5b6012d55f731eef11ab7f880b0e85) [(gdas_snowanl.log)](https://gist.github.com/emcbot/4e47c59cb45fcac71365c35ac69ab38c) [(gfs_snowanl.log)](https://gist.github.com/emcbot/6f618884e7b90d6a78f7ffb0a552322f)
author_association: NONE
user: login: emcbot
created_at: 2025-05-15T19:19:52Z
body: Experiment C96C48mx500_S2SW_cyc_gfs **FAILED** on Gaeac6 in Build# 1 in
`/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/EXPDIR/C96C48mx500_S2SW_cyc_gfs_e08c9b84`
author_association: NONE
user: login: emcbot
created_at: 2025-05-15T19:25:41Z
body: Experiment C96C48_hybatmaerosnowDA **FAILED** on Gaeac6 in Build# 1 with error logs:
```
/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_e08c9b84/logs/2021122018/enkfgdas_esnowanl.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_e08c9b84/logs/2021122018/gdas_snowanl.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_e08c9b84/logs/2021122018/gfs_snowanl.log
```

Follow link here to view the contents of the above file(s): [(enkfgdas_esnowanl.log)](https://gist.github.com/emcbot/aee8ef961488b063a96d8e2da2295ee6) [(gdas_snowanl.log)](https://gist.github.com/emcbot/5302f0783317cf59be8f96fee49f26a5) [(gfs_snowanl.log)](https://gist.github.com/emcbot/e504c70409965b07f46aca0d73f145ae)
author_association: NONE
user: login: emcbot
created_at: 2025-05-15T19:25:52Z
body: Experiment C96C48_hybatmaerosnowDA **FAILED** on Gaeac6 in Build# 1 in
`/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_e08c9b84`
author_association: NONE
user: login: emcbot
created_at: 2025-05-15T19:30:34Z
body: Experiment C96C48_hybatmaerosnowDA **FAILED** on Hera in Build# 2 with error logs:
```
/scratch1/NCEPDEV/global/glopara/CI/3642/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_e08c9b84/logs/2021122018/enkfgdas_esnowanl.log
/scratch1/NCEPDEV/global/glopara/CI/3642/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_e08c9b84/logs/2021122018/gdas_snowanl.log
/scratch1/NCEPDEV/global/glopara/CI/3642/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_e08c9b84/logs/2021122018/gfs_snowanl.log
```

Follow link here to view the contents of the above file(s): [(enkfgdas_esnowanl.log)](https://gist.github.com/emcbot/8677fc7b7adcda6675cacd47dcee6b1c) [(gdas_snowanl.log)](https://gist.github.com/emcbot/6a80e21f6be580a88b96112fc73ed4ce) [(gfs_snowanl.log)](https://gist.github.com/emcbot/397336b52dd785b75de208312a54303d)
author_association: NONE
user: login: emcbot
created_at: 2025-05-15T19:30:46Z
body: Experiment C96C48_hybatmaerosnowDA **FAILED** on Hera in Build# 2 in
`/scratch1/NCEPDEV/global/glopara/CI/3642/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_e08c9b84`
author_association: NONE
user: login: emcbot
created_at: 2025-05-15T19:31:11Z
body: Experiment C96C48mx500_S2SW_cyc_gfs **FAILED** on Hera in Build# 2 with error logs:
```
/scratch1/NCEPDEV/global/glopara/CI/3642/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_e08c9b84/logs/2021122018/enkfgdas_esnowanl.log
/scratch1/NCEPDEV/global/glopara/CI/3642/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_e08c9b84/logs/2021122018/enkfgfs_esnowanl.log
/scratch1/NCEPDEV/global/glopara/CI/3642/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_e08c9b84/logs/2021122018/gdas_snowanl.log
/scratch1/NCEPDEV/global/glopara/CI/3642/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_e08c9b84/logs/2021122018/gfs_snowanl.log
```

Follow link here to view the contents of the above file(s): [(enkfgdas_esnowanl.log)](https://gist.github.com/emcbot/5b745dfd80fa34b7b4248042610034d7) [(enkfgfs_esnowanl.log)](https://gist.github.com/emcbot/3dab40c10b53ca16fc0a693b7bf94b77) [(gdas_snowanl.log)](https://gist.github.com/emcbot/9739f823ad792758356b69bbd7264146) [(gfs_snowanl.log)](https://gist.github.com/emcbot/8af070908df23538759f7426169e940b)
author_association: NONE
user: login: emcbot
created_at: 2025-05-15T19:31:26Z
body: Experiment C96C48mx500_S2SW_cyc_gfs **FAILED** on Hera in Build# 2 in
`/scratch1/NCEPDEV/global/glopara/CI/3642/RUNTESTS/EXPDIR/C96C48mx500_S2SW_cyc_gfs_e08c9b84`
author_association: NONE
user: login: emcbot
created_at: 2025-05-15T19:55:09Z
body: Experiment C48mx500_3DVarAOWCDA **FAILED** on Hera in Build# 2 with error logs:
```
/scratch1/NCEPDEV/global/glopara/CI/3642/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_e08c9b84/logs/2021032500/gfs_fcst_seg0.log
```

Follow link here to view the contents of the above file(s): [(gfs_fcst_seg0.log)](https://gist.github.com/emcbot/7932a9fbd262a27703874e262f0c0469)
author_association: NONE
user: login: emcbot
created_at: 2025-05-15T19:55:19Z
body: Experiment C48mx500_3DVarAOWCDA **FAILED** on Hera in Build# 2 in
`/scratch1/NCEPDEV/global/glopara/CI/3642/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_e08c9b84`
author_association: NONE
user: login: emcbot
created_at: 2025-05-15T20:06:54Z
body: Experiment C96C48_ufs_hybatmDA **FAILED** on Hera in Build# 2 with error logs:
```
/scratch1/NCEPDEV/global/glopara/CI/3642/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_e08c9b84/logs/2024022400/enkfgdas_atmensanlobs.log
/scratch1/NCEPDEV/global/glopara/CI/3642/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_e08c9b84/logs/2024022400/gdas_atmanlvar.log
/scratch1/NCEPDEV/global/glopara/CI/3642/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_e08c9b84/logs/2024022400/gfs_atmanlvar.log
```

Follow link here to view the contents of the above file(s): [(enkfgdas_atmensanlobs.log)](https://gist.github.com/emcbot/4410a4900a3630b4c8543192e1952e9e) [(gdas_atmanlvar.log)](https://gist.github.com/emcbot/3ffe59b303ea9b008b91c2cf4275508f) [(gfs_atmanlvar.log)](https://gist.github.com/emcbot/5e2c9ea1258e9fbf2066d67671805693)
author_association: NONE
user: login: emcbot
created_at: 2025-05-15T20:07:05Z
body: Experiment C96C48_ufs_hybatmDA **FAILED** on Hera in Build# 2 in
`/scratch1/NCEPDEV/global/glopara/CI/3642/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_e08c9b84`
author_association: NONE
user: login: DavidNew-NOAA
created_at: 2025-05-16T01:12:15Z
body: Hold off on CI. There appears to be an issue in which the obs spaces for the atm variational da yaml is made empty. Will work through it tomorrow
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-16T14:17:44Z
body: @DavidNew-NOAA :  It looks like the atm_init job creates `atmanlvar.yaml` before it stages the obs to the run directory.  Thus, the check in `jedi.py` does not find any observations.  

As a test, I installed `DavidNew-NOAA:feature/marine-analysis-jedi-class` on Hera.  I made the following change in `ush/python/pygfs/task/atm_analysis.py`
```
--- a/ush/python/pygfs/task/atm_analysis.py
+++ b/ush/python/pygfs/task/atm_analysis.py
@@ -99,8 +99,8 @@ class AtmAnalysis(Task):
         """

         # initialize JEDI variational application
-        logger.info(f"Initializing JEDI variational DA application")
-        self.jedi_dict['atmanlvar'].initialize(self.task_config)
+        ##logger.info(f"Initializing JEDI variational DA application")
+        ##self.jedi_dict['atmanlvar'].initialize(self.task_config)

         # initialize JEDI FV3 increment conversion application
         logger.info(f"Initializing JEDI FV3 increment conversion application")
@@ -167,6 +167,10 @@ class AtmAnalysis(Task):
         ]
         FileHandler({'mkdir': newdirs}).sync()

+        # initialize JEDI variational application
+        logger.info(f"Initializing JEDI variational DA application")
+        self.jedi_dict['atmanlvar'].initialize(self.task_config)
+
```
I moved creation of the yaml to the end of the initialize method.    

I ran `test_gdasapp_atm_jjob_var_init`.  The test passed.  The `atmanlvar.yaml` does not have an empty `obs space`.

Execution of test_gdasapp_atm_jjob_var_run passed.  

Could this explain why C96C48_ufs_hybatmDA failed?   Could a similar problem arise in JEDI aerosol or snow DA?
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-16T14:21:19Z
body: Thanks @RussTreadon-NOAA , that makes perfect sense. I was foolishly trying to debug the `jedi.py` code, when the problem was the `atm_analysis.py`. And yes, surely the same problem will arise with snow and aerosol, looking that the code.
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-16T14:29:39Z
body: Retesting ` C96C48_ufs_hybatmDA`
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-16T16:02:04Z
body: @DavidNew-NOAA :  We have a new problem.

 We run the letkf in two steps for atmospheric DA: 
1. observer - job atmensanlobs
2. solver - job atmensanlsol

The atmensanlinit job creates `atmensanlobs.yaml` for the observer and `atmensanlsol.yaml` for the solver.

With your recent changes the observations are present in the run directory when we create `atmensanlobs.yaml`.  Thus, `observers:` is populated in `atmensanlsol.yaml`.  

The solver job reads the diagnostic files from the observer job.  These files do not exist because atmensanlobs has not yet run.  As a result, `atmensanlsol.yaml` winds up with
```
observations:
  observers: []
```

The atmensanlsol job aborts when it runs.
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-16T17:06:47Z
body: @DavidNew-NOAA .  The `# Naming conventions for observation files` section of `jcb-prototype_lgetkf_observer.yaml.j2` and `jcb-prototype_lgetkf_solver.yaml.j2` in `parm/atm` differ as follows

`jcb-prototype_lgetkf_observer.yaml.j2` has
```
# Naming conventions for observation files
atmosphere_obsdataout_prefix: diag_lobs_

```

whereas `jcb-prototype_lgetkf_solver.yaml.j2` has
```
# Naming conventions for observation files
atmosphere_obsdatain_path: "{{atm_obsdataout_path}}"
atmosphere_obsdatain_prefix: diag_lobs_
atmosphere_obsdatain_suffix: "_{{ current_cycle | to_YMDH }}.nc"
atmosphere_obsdataout_prefix: diag_solv_

```

The solver template points `atmosphere_obsdatain_path` at `atm_obsdataout_path`.   The obsdata_prefix is also changed so that it looks for `diag_lobs_`.  

Yamls from the 20250515 run of stable-nightly are still  on Hera.  Look in `/scratch1/NCEPDEV/da/Russ.Treadon/CI/GDASApp/stable/20250515/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci/C96C48_ufs_hybatmDA/COMROOT/C96C48_ufs_hybatmDA/enkfgdas.20240224/00/ensstat/analysis/atmos/`

Compare `enkfgdas.t00z.atmensanlobs.yaml` and `enkfgdas.t00z.atmensanlsol.yaml`.  Here's a sampling of what you'll see
```
77c77
<         name: RoundRobin
---
>         name: Halo
82c82
<           obsfile: ./obs//gdas.t00z.aircraft.tm00.nc
---
>           obsfile: ./diags//diag_lobs_aircraft_2024022400.nc
91c91
<           obsfile: ./diags//diag_lobs_aircraft_2024022400.nc
---
>           obsfile: ./diags//diag_solv_aircraft_2024022400.nc
```
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-16T17:11:57Z
body: @RussTreadon-NOAA Ah! I did not think to look at the JCB prototype YAMLs! OK, well that points to the solution which we discuss which is to initialize the solver after we run the observer. Working on it
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-16T18:51:16Z
body: Testing `C96C48_hybatmaerosnowDA` now
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-16T20:16:13Z
body: After running the `C96C48_hybatmaerosnowDA` case, the `gdas_anlstat` job failed. The reason is that the `observations` key in the `snow.yaml` input yaml is `null`. I will have the investigate the reason for this Monday.
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-18T20:14:18Z
body: **GDASApp stable-nightly testing**

GDASApp stable-nightly testing was modified to use `DavidNew-NOAA:feature/marine-analysis-jedi-class` as the g-w branch in which GDASApp `develop` was built and run.  stable-nightly locally updated the following JEDI hashes
```
Hera(hfe02):/scratch1/NCEPDEV/da/Russ.Treadon/CI/GDASApp/stable/20250518/global-workflow/sorc/gdas.cd$ git diff sorc
diff --git a/sorc/ioda b/sorc/ioda
index 1b4484b..23ad14f 160000
--- a/sorc/ioda
+++ b/sorc/ioda
@@ -1 +1 @@
-Subproject commit 1b4484bf17d942889cd36592d0e83fab3c8a8421
+Subproject commit 23ad14f75bd057ed5a2160d876b9c6fcb45f32ab
diff --git a/sorc/iodaconv b/sorc/iodaconv
index a56eef6..5a445d4 160000
--- a/sorc/iodaconv
+++ b/sorc/iodaconv
@@ -1 +1 @@
-Subproject commit a56eef6bdca700b4356eb3b3608af642e880549d
+Subproject commit 5a445d4988c14e81be8dfc465441dbabbfcf009d
diff --git a/sorc/saber b/sorc/saber
index 8009aed..e76dc63 160000
--- a/sorc/saber
+++ b/sorc/saber
@@ -1 +1 @@
-Subproject commit 8009aed05836111174d62c4b4298be524efa343a
+Subproject commit e76dc63e988c3883b4e9942afe224f098b498948
diff --git a/sorc/ufo b/sorc/ufo
index 7cbaae3..8c44fa3 160000
--- a/sorc/ufo
+++ b/sorc/ufo
@@ -1 +1 @@
-Subproject commit 7cbaae3e03624829992491b35cbe3a8652650885
+Subproject commit 8c44fa3edec46c68ed137a398db3df3821d3a8a4
``` 

GDASApp successfully built.  All GDASApp ctests passed except one
```
The following tests FAILED:
        2090 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlletkf_202103250000 (Failed)
```

A check of `gdas_marineanlletkf.log` found
```
  File "/scratch1/NCEPDEV/da/Russ.Treadon/CI/GDASApp/stable/20250518/global-workflow/parm/gdas/soca/jcb-base.yaml.j2", line 78, in top-level template code
    marine_obsdatain_suffix: '.{{ PDY | to_YMD }}{{ "%02d"| format(cyc) }}.nc4'
  File "/scratch1/NCEPDEV/da/Russ.Treadon/CI/GDASApp/stable/20250518/global-workflow/ush/python/wxflow/jinja.py", line 141, in <lambda>
    env.filters["to_YMD"] = lambda dt: to_YMD(dt) if not isinstance(dt, SilentUndefined) else dt
  File "/scratch1/NCEPDEV/da/Russ.Treadon/CI/GDASApp/stable/20250518/global-workflow/ush/python/wxflow/timetools.py", line 150, in datetime_to_YMD
    raise Exception(f"Bad datetime: '{dt}'")
Exception: Bad datetime: '20210325'
```
Make the following change to `gdas.cd/parm/soca/jcb-base.yaml.j2` based on `gdas.cd/parm/atm/`
```
@@ -75,10 +75,10 @@ marine_obsdatastats_path: '{{COMOUT_OCEAN_ANALYSIS}}/diags'

 marine_obsdatain_path: '{{DATA}}/obs'
 marine_obsdatain_prefix: '{{OPREFIX}}'
-marine_obsdatain_suffix: '.{{ PDY | to_YMD }}{{ "%02d"| format(cyc) }}.nc4'
+marine_obsdatain_suffix: ".{{ current_cycle | to_YMDH }}.nc4"
 marine_obsdataout_path: '{{DATA}}/diags'
 marine_obsdataout_prefix: ''
-marine_obsdataout_suffix: '.{{ PDY | to_YMD }}{{ "%02d"| format(cyc) }}.nc4'
+marine_obsdataout_suffix: ".{{ current_cycle | to_YMDH }}.nc4"

 # Outer loops
 marine_ninner_1: '{{SOCA_NINNER}}'
```

Rewind and rerun failed job.  gdas_marineanlletkf ran to completion.   

Not sure this is the correct fix, but it worked.  Whatever the correct fix is, we need to update GDASApp and then update the `sorc/gdas.cd` hash in `DavidNew-NOAA:feature/marine-analysis-jedi-class`
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-19T12:47:12Z
body: @RussTreadon-NOAA I think it may have something do with this line https://github.com/DavidNew-NOAA/global-workflow/blob/edad0e1d74b84e32a7dc49935d0dd7bc060bfc01/ush/python/pygfs/task/marine_letkf.py#L58 and the one below it. I removed similar lines in `marine_analysis.py` here https://github.com/NOAA-EMC/global-workflow/blob/f46f47967b81b1e929868a38b94d28314e9b9fc9/ush/python/pygfs/task/marine_analysis.py#L210 . These lines are actually redefining `cyc` and `PDY`, because they should already exist in `task_config` if they're environment variables. I suspect the call to `os.getenv()` vs this line https://github.com/NOAA-EMC/global-workflow/blob/f46f47967b81b1e929868a38b94d28314e9b9fc9/scripts/exglobal_marine_analysis_letkf.py#L18 which loads all the environment variables into the `config` variables (which becomes the basis for `task_config`) does something different, because `cast_strdict_as_dtypedict()` from `wxflow` does some special magic.

All this being said, could you try just commenting out lines 58 and 59 and re-running the test and see what happens? I suspect this will fix the problem
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-19T13:15:51Z
body: @DavidNew-NOAA , the following changes were made in my Hera working copy
- revert `gdas.cd/parm/soca/jcb-base.yaml.j2` back to what is in [gdas.cd @ cdc0945](https://github.com/NOAA-EMC/GDASApp/tree/cdc0945b60a7cde2a5f2e04e793d38f7d96c720a)
- comment out lines 58 and 59 of `ush/python/pygfs/task/marine_letkf.py`

The C48mx500_hybAOWCDA.xml 20210325 00Z gdas_marineanlletkf was rewound and rebooted.  The job ran to completion with the above local modifications.  Thus, this solution also yields a successful run.

A check of `obsdatain_suffix` and `obsdataout_suffix` shows marine to be outlier.  The aero, atm, and snow variables in their respective `jcb-base.yaml.j2` all follow the pattern
```
[aero, atm, snow]_obsdatain_suffix: ".tm00.nc"
[aero, atm, snow]_obsdataout_suffix: "_{{ current_cycle | to_YMDH }}.nc"
```
whereas marine has
```
marine_obsdatain_suffix: '.{{ PDY | to_YMD }}{{ "%02d"| format(cyc) }}.nc4'
marine_obsdataout_suffix: '.{{ PDY | to_YMD }}{{ "%02d"| format(cyc) }}.nc4'
```

Do we keep the soca `jcb-base.yaml.j2` as is or modify it to be more consistent with `jcb-base.yaml.j2` for other DA types?
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-19T13:28:00Z
body: @RussTreadon-NOAA Thanks for testing! Yes, we should definitely change the marine jcb base to make it consistent with the others. That `to_YMDH` filter is there for a reason.
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-19T13:56:41Z
body: @RussTreadon-NOAA I have a solution for the `gdas_anlstat` failure I'm working on thanks to @CoryMartin-NOAA 's help base. It will require https://github.com/NOAA-EMC/jcb-gdas/pull/123 to be merged, and then I can make a GDASApp PR which update the jcb-gdas hash and also makes the jcb-base changes you suggested. Testing now
author_association: CONTRIBUTOR
user: login: danholdaway
created_at: 2025-05-19T14:22:33Z
body: When the JEDI runs with no obs spaces does the code crash or fail on checking to references?
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-19T14:49:19Z
body: @danholdaway I have not tested this possibility, but I can
author_association: CONTRIBUTOR
user: login: danholdaway
created_at: 2025-05-19T16:28:09Z
body: > @danholdaway I have not tested this possibility, but I can

Isn't this what happened in the test @RussTreadon-NOAA reported on? Perhaps the logs reveal what happened.
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-19T16:34:03Z
body: @danholdaway , earlier versions of this PR had
```
observations:
  observers: []
```
`gdas.x` aborted when run in letkf solver mode with the above `observers`.  Is this your question?

_OR_

Are asking about the situation in which the `observers` section is complete in the yaml read by `gdas.x`, _but_ the input obsfile is missing or zero length?

author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-20T00:19:56Z
body: @aerorahul @KateFriedman-NOAA @TerrenceMcGuinness-NOAA I've re-tested this PR, and I'm sufficiently confident that it is ready for CI testing.
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-20T11:56:25Z
body: **g-w CI**

`DavidNew-NOAA:feature/marine-analysis-jedi-class` at 5d789749 installed on Cactus, Hera, Hercules, and Orion.  g-w CI configured and initiated on each machine.  g-w CI results will be posted in this PR as jobs complete on each machine.

author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-20T12:00:43Z
body: Thanks @RussTreadon-NOAA 
author_association: CONTRIBUTOR
user: login: emcbot
created_at: 2025-05-20T13:26:16Z
body: Experiment C48mx500_3DVarAOWCDA **FAILED** on Gaeac6 in Build# 1 with error logs:
```
/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_5d789749/logs/2021032500/gdas_marineanlchkpt.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_5d789749/logs/2021032500/gfs_marineanlchkpt.log
```

Follow link here to view the contents of the above file(s): [(gdas_marineanlchkpt.log)](https://gist.github.com/emcbot/ce5b59752b289f990c4f66fcd748521e) [(gfs_marineanlchkpt.log)](https://gist.github.com/emcbot/3f696b134890dcee0f8b54f8b9b95612)
author_association: NONE
user: login: emcbot
created_at: 2025-05-20T13:26:26Z
body: Experiment C48mx500_3DVarAOWCDA **FAILED** on Gaeac6 in Build# 1 in
`/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_5d789749`
author_association: NONE
user: login: emcbot
created_at: 2025-05-20T13:32:06Z
body: Experiment C48mx500_hybAOWCDA **FAILED** on Gaeac6 in Build# 1 with error logs:
```
/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/COMROOT/C48mx500_hybAOWCDA_5d789749/logs/2021032500/gdas_marineanlchkpt.log
```

Follow link here to view the contents of the above file(s): [(gdas_marineanlchkpt.log)](https://gist.github.com/emcbot/4bd66eda378b46495465d544a0f6b257)
author_association: NONE
user: login: emcbot
created_at: 2025-05-20T13:32:14Z
body: Experiment C48mx500_hybAOWCDA **FAILED** on Gaeac6 in Build# 1 in
`/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/EXPDIR/C48mx500_hybAOWCDA_5d789749`
author_association: NONE
user: login: emcbot
created_at: 2025-05-20T13:32:24Z
body: Experiment C96C48mx500_S2SW_cyc_gfs **FAILED** on Gaeac6 in Build# 1 with error logs:
```
/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_5d789749/logs/2021122018/gdas_marineanlchkpt.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_5d789749/logs/2021122018/gfs_marineanlchkpt.log
```

Follow link here to view the contents of the above file(s): [(gdas_marineanlchkpt.log)](https://gist.github.com/emcbot/3da39e412fb8a748efc661fdfe09065c) [(gfs_marineanlchkpt.log)](https://gist.github.com/emcbot/d0c4f0e235080b77e734876d18ffb6fe)
author_association: NONE
user: login: emcbot
created_at: 2025-05-20T13:32:35Z
body: Experiment C96C48mx500_S2SW_cyc_gfs **FAILED** on Gaeac6 in Build# 1 in
`/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/EXPDIR/C96C48mx500_S2SW_cyc_gfs_5d789749`
author_association: NONE
user: login: DavidNew-NOAA
created_at: 2025-05-20T13:39:00Z
body: @aerorahul Sorry sorry sorry. I only modified HERA.env out of laziness when I first worked on this PR. My fault
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-20T13:47:50Z
body: OK fixed
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-20T14:07:58Z
body: **Hera g-w CI**

g-w CI is still running on Hera but the following jobs are DEAD
```
/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C48mx500_3DVarAOWCDA_pr3642
202103250000          gdas_arch_tars                    10974296                DEAD                  15         2          41.0

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C48mx500_hybAOWCDA_pr3642
202103250000          gdas_arch_tars                    10973886                DEAD                  15         2          41.0

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/C96C48mx500_S2SW_cyc_gfs_pr3642
202112201800    enkfgfs_earc_tars_00                    10972891                DEAD                  15         2          27.0
202112201800          gdas_arch_tars                    10974204                DEAD                  15         2          41.0
```

A check of the log files find similar `FileNotFoundError`

_C48mx500_3DVarAOWCDA and C48mx500_hybAOWCDA_
```
FileNotFoundError: FATAL ERROR: Required file, directory, or glob gdas.20210325/00/analysis/ocean/gdas.t00z.ocnana.nc not found!
```

_C96C48mx500_S2SW_cyc_gfs_
```
FileNotFoundError: FATAL ERROR: Required file, directory, or glob gdas.20211220/18/analysis/ocean/gdas.t18z.ocnana.nc not found!
FileNotFoundError: FATAL ERROR: Required file, directory, or glob enkfgfs.20211220/18/ensstat/analysis/ocean/enkfgdas.ocean.t18z.bg_ensvar.nc not found!
```
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-20T14:41:06Z
body: @RussTreadon-NOAA I'm actually a little perplexed as to where the `ocnana` file is created. Does it exist the rundir for the marine analysis job in your test? 

The `bg_ensvar` I'm even a bit more perplexed about. It's created here https://github.com/NOAA-EMC/GDASApp/blob/5aac81fe2feab69c5cccbc0b5bb8c47d369c1875/ush/soca/marine_recenter.py#L274 with a `Filehandler` `copy` call, but that call should fail if they file does not exist.
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-20T14:46:45Z
body: @DavidNew-NOAA 

For Hera gdas_arch_tars failures, there is a discrepancy in filenames.  The gdas COMROOT directories contain `ocn.ana.nc`,  **_not_** `ocnana.nc`.
```
/scratch1/NCEPDEV/stmp2/Russ.Treadon/COMROOT/C96C48mx500_S2SW_cyc_gfs_pr3642/gdas.20211220/18/analysis/ocean
  -rw-rw-r--+ 1 Russ.Treadon stmp  162311 May 20 12:16 gdas.t18z.ice.ana.nc
  -rw-rw-r--+ 1 Russ.Treadon stmp   87158 May 20 12:16 gdas.t18z.ice.incr.nc
  -rw-rw-r--+ 1 Russ.Treadon stmp   87151 May 20 12:26 gdas.t18z.ice.incr.postproc.nc
  -rw-rw-r--+ 1 Russ.Treadon stmp 1565229 May 20 12:16 gdas.t18z.ocn.ana.nc
  -rw-rw-r--+ 1 Russ.Treadon stmp 1056675 May 20 12:16 gdas.t18z.ocn.incr.nc
  -rw-rw-r--+ 1 Russ.Treadon stmp 2585207 May 20 12:26 gdas.t18z.ocninc.nc
```

For the enkfgfs_earc_tars_00 failure, there is a dump discrepancy.  The gfs job is looking for a gdas file
```
FATAL ERROR: Required file, directory, or glob enkfgfs.20211220/18/ensstat/analysis/ocean/enkfgdas.ocean.t18z.bg_ensvar.nc not found!
```
A check of COMROOT shows that
```
/scratch1/NCEPDEV/stmp2/Russ.Treadon/COMROOT/C96C48mx500_S2SW_cyc_gfs_pr3642/enkfgfs.20211220/18/ensstat/analysis/ocean
  -rw-rw-r--+ 1 Russ.Treadon stmp 2049599 May 20 12:21 enkfgfs.ocean.t18z.bg_ensvar.nc
```
exists.

The enkfgfs job should look for `enkfgfs`, **_not_** `enkfgdas`.
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-20T15:01:58Z
body: @DavidNew-NOAA and @DavidHuber-NOAA 

`parm/archive/enkf.yaml.j2` contains
```
        {% if DOHYBVAR_OCN %}
        - "{{ COMIN_OCEAN_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head_ocean }}bg_ensvar.nc"
        - "{{ COMIN_ICE_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head_ice }}bg_ensvar.nc"
        {% endif %}
```

`parm/archive/master_enkf.yaml.j2` sets
```
# TODO Should enkfgdas be replaced with RUN?
{% set head_ocean = "enkfgdas.ocean.t" + cycle_HH + "z." %}
{% set head_ice = "enkfgdas.ice.t" + cycle_HH + "z." %}
````

Looks like it's time to act on the `TODO`
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-20T15:04:32Z
body: @RussTreadon-NOAA The TODO is being fixed in this PR https://github.com/NOAA-EMC/global-workflow/pull/3710
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-20T15:11:06Z
body: Thank you @DavidNew-NOAA .  We'll need to merge these PRs if we want g-w CI to pass. Alternatively we accept the failures as _explainable_.  My preference is to combine PRs so g-w CI passes.  What do you think @DavidHuber-NOAA?

Still poking around trying to understand the reason for the `ocnana` versus `ocn.ana` difference.
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-20T15:13:59Z
body: The `ocnana.nc` reference comes form `parm/archive/gdasocean_analysis.yaml.j2`
```
    name: "GDASOCEAN_ANALYSIS"
    target: "{{ ATARDIR }}/{{ cycle_YMDH }}/gdasocean_analysis.tar"
    required:
        # analysis and analysis increments
        - '{{ COMIN_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}ocninc.nc'
        {% for domain in ["ocn", "ice"] %}
        - '{{ COMIN_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}{{domain}}.incr.nc'
        - '{{ COMIN_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}{{domain}}ana.nc'
        {% endfor %}
```
Looks like we need to add a period between `{{domain}}` and `ana`.  That is,

```
        - '{{ COMIN_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}{{domain}}.ana.nc'
```
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-20T15:17:21Z
body: Thank you @shlyaeva ! I'll make a quick bugfix gdasapp PR rather than change the archiving script in GW
author_association: CONTRIBUTOR
user: login: shlyaeva
created_at: 2025-05-20T15:20:14Z
body: I'd like to make sure the tests are passing before merging this PR, since it's a reasonably big change (for the better code!) that required a lot of bookkeeping wrt to filenames. I would also like to make sure that the second cycle can start correctly with these changes (e.g. ensure that the cice restart and mom6 increment are in correct directories), is this covered by the tests?
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-20T15:24:23Z
body: @DavidNew-NOAA and @shlyaeva 

Adding the dot helps, but we do not have `*.csv` files in COMROOT.   `GDASOCEAN_ANALYSIS` from  `parm/archive/gdasocean_analysis.yaml.j2` includes `.csv`
```
        # obs space diags
        - '{{ COMIN_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}ocn.*.stats.csv'
        - '{{ COMIN_OCEAN_ANALYSIS | relpath(ROTDIR) }}/diags/*.nc4'
```
There are no csv files in COMROOT.  Therefore the archive job fails. 
```
    raise FileNotFoundError(f"FATAL ERROR: Required file, directory, or glob {item} not found!")
FileNotFoundError: FATAL ERROR: Required file, directory, or glob gdas.20210325/00/analysis/ocean/gdas.t00z.ocn.*.stats.csv not found!
+ JGLOBAL_ARCHIVE_TARS[80]export err=1
```

Did the csv files get removed by a GDASApp PR? 
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-20T15:44:45Z
body: @RussTreadon-NOAA Scratch my last comment. This line should be moving the csv files to COM:

https://github.com/DavidNew-NOAA/global-workflow/blob/7024c677cd6ea38db86fb8b7c792d931b3c8ed63/ush/python/pygfs/task/marine_analysis.py#L205
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-20T17:21:56Z
body: 


> @DavidNew-NOAA and @shlyaeva
> 
> Adding the dot helps, but we do not have `*.csv` files in COMROOT. `GDASOCEAN_ANALYSIS` from `parm/archive/gdasocean_analysis.yaml.j2` includes `.csv`
> 
> ```
>         # obs space diags
>         - '{{ COMIN_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}ocn.*.stats.csv'
>         - '{{ COMIN_OCEAN_ANALYSIS | relpath(ROTDIR) }}/diags/*.nc4'
> ```
> 
> There are no csv files in COMROOT. Therefore the archive job fails.
> 
> ```
>     raise FileNotFoundError(f"FATAL ERROR: Required file, directory, or glob {item} not found!")
> FileNotFoundError: FATAL ERROR: Required file, directory, or glob gdas.20210325/00/analysis/ocean/gdas.t00z.ocn.*.stats.csv not found!
> + JGLOBAL_ARCHIVE_TARS[80]export err=1
> ```
> 
> Did the csv files get removed by a GDASApp PR?

@RussTreadon-NOAA I _think_ I figured out the csv issue.

It should be:

`- '{{ COMIN_OCEAN_ANALYSIS | relpath(ROTDIR) }}/diags/{{ head }}ocn.*.stats.csv'`

now, instead of 

`- '{{ COMIN_OCEAN_ANALYSIS | relpath(ROTDIR) }}/{{ head }}ocn.*.stats.csv'`

I _think_. Testing now
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-20T17:36:10Z
body: @RussTreadon-NOAA Odd. It seems like soca_diag_stats code is not producing any csv files
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-20T18:06:02Z
body: **Hercules g-w CI**

All jobs in all cases run on Hercules completed.  Note that Hercules g-w CI runs with `DO_ARCHCOM: 'NO'`.  Thus, we do not see archive job failures on Hercules.

```
/work2/noaa/stmp/rtreadon/HERCULES/EXPDIR/C48_ATM_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103231200        Done    May 20 2025 11:45:03    May 20 2025 12:55:02

/work2/noaa/stmp/rtreadon/HERCULES/EXPDIR/C48_S2SW_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103231200        Done    May 20 2025 11:45:05    May 20 2025 12:55:04

/work2/noaa/stmp/rtreadon/HERCULES/EXPDIR/C48_S2SWA_gefs_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103231200        Done    May 20 2025 11:45:04    May 20 2025 13:15:03

/work2/noaa/stmp/rtreadon/HERCULES/EXPDIR/C48mx500_3DVarAOWCDA_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103241800        Done    May 20 2025 11:45:05    May 20 2025 12:10:07
202103250000        Done    May 20 2025 11:45:05    May 20 2025 15:30:05

/work2/noaa/stmp/rtreadon/HERCULES/EXPDIR/C48mx500_hybAOWCDA_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103241800        Done    May 20 2025 11:45:06    May 20 2025 12:15:07
202103250000        Done    May 20 2025 11:45:06    May 20 2025 15:05:07

/work2/noaa/stmp/rtreadon/HERCULES/EXPDIR/C96C48_hybatmDA_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201800        Done    May 20 2025 11:45:07    May 20 2025 12:15:08
202112210000        Done    May 20 2025 11:45:07    May 20 2025 14:50:08
202112210600        Done    May 20 2025 11:45:07    May 20 2025 15:15:07

/work2/noaa/stmp/rtreadon/HERCULES/EXPDIR/C96C48_hybatmaerosnowDA_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201200        Done    May 20 2025 11:45:08    May 20 2025 12:15:09
202112201800        Done    May 20 2025 11:45:08    May 20 2025 14:45:09
202112210000        Done    May 20 2025 11:45:08    May 20 2025 15:05:08

/work2/noaa/stmp/rtreadon/HERCULES/EXPDIR/C96C48_ufs_hybatmDA_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202402231800        Done    May 20 2025 11:45:09    May 20 2025 12:15:10
202402240000        Done    May 20 2025 11:45:09    May 20 2025 15:55:08
202402240600        Done    May 20 2025 11:45:09    May 20 2025 16:05:09

/work2/noaa/stmp/rtreadon/HERCULES/EXPDIR/C96C48mx500_S2SW_cyc_gfs_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201200        Done    May 20 2025 11:45:10    May 20 2025 12:15:11
202112201800        Done    May 20 2025 11:45:10    May 20 2025 16:45:10
202112210000        Done    May 20 2025 11:45:10    May 20 2025 17:15:10
202112211800        Done    May 20 2025 12:20:11    May 20 2025 17:10:10

/work2/noaa/stmp/rtreadon/HERCULES/EXPDIR/C96_atm3DVar_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201800        Done    May 20 2025 11:45:11    May 20 2025 12:15:12
202112210000        Done    May 20 2025 11:45:11    May 20 2025 14:55:13
202112210600        Done    May 20 2025 11:45:11    May 20 2025 15:05:11

/work2/noaa/stmp/rtreadon/HERCULES/EXPDIR/C96mx100_S2S_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
199405010000        Done    May 20 2025 11:45:12    May 20 2025 14:00:12
```
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-20T18:13:34Z
body: **Orion g-w CI**

All jobs in all cases run on Orion completed. Note that Orion g-w CI runs with `DO_ARCHCOM: 'NO'`. Thus, we do not see archive job failures on Orion.

```
/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C48_ATM_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103231200        Done    May 20 2025 12:15:10    May 20 2025 14:00:05

/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C48mx500_3DVarAOWCDA_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103241800        Done    May 20 2025 12:15:12    May 20 2025 12:35:07
202103250000        Done    May 20 2025 12:15:12    May 20 2025 16:45:06

/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C48mx500_hybAOWCDA_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103241800        Done    May 20 2025 12:15:14    May 20 2025 12:35:08
202103250000        Done    May 20 2025 12:15:14    May 20 2025 15:15:08

/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C48_S2SWA_gefs_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103231200        Done    May 20 2025 12:15:15    May 20 2025 14:00:15

/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C48_S2SW_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202103231200        Done    May 20 2025 12:15:17    May 20 2025 14:05:11

/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C96_atm3DVar_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201800        Done    May 20 2025 12:15:19    May 20 2025 12:35:14
202112210000        Done    May 20 2025 12:15:19    May 20 2025 15:55:12
202112210600        Done    May 20 2025 12:15:19    May 20 2025 15:20:13

/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C96C48_hybatmaerosnowDA_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201200        Done    May 20 2025 12:15:21    May 20 2025 12:40:16
202112201800        Done    May 20 2025 12:15:21    May 20 2025 15:55:14
202112210000        Done    May 20 2025 12:15:21    May 20 2025 15:30:15

/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C96C48_hybatmDA_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201800        Done    May 20 2025 12:15:22    May 20 2025 12:35:22
202112210000        Done    May 20 2025 12:15:22    May 20 2025 15:55:15
202112210600        Done    May 20 2025 12:15:22    May 20 2025 15:30:16

/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C96C48mx500_S2SW_cyc_gfs_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202112201200        Done    May 20 2025 12:15:25    May 20 2025 12:40:20
202112201800        Done    May 20 2025 12:15:25    May 20 2025 17:15:18
202112210000        Done    May 20 2025 12:15:25    May 20 2025 18:01:24
202112211800        Done    May 20 2025 12:45:19    May 20 2025 18:09:21

/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C96C48_ufs_hybatmDA_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202402231800        Done    May 20 2025 12:15:26    May 20 2025 12:40:22
202402240000        Done    May 20 2025 12:15:26    May 20 2025 17:25:20
202402240600        Done    May 20 2025 12:15:26    May 20 2025 16:50:22

/work2/noaa/stmp/rtreadon/ORION/EXPDIR/C96mx100_S2S_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED
199405010000        Done    May 20 2025 12:15:28    May 20 2025 14:20:24
```
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-20T18:20:23Z
body: @RussTreadon-NOAA I think I found our csv error. Working on it
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-21T01:17:28Z
body: **WCOSS2 (Cactus) g-w CI**

All jobs in all cases, except C96_atm3DVar_extended, run on Cactus completed. Note that Cactus g-w CI runs with `DO_ARCHCOM: 'NO'`. Thus, we do not see archive job failures on Cactus.

```
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_ATM_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    May 20 2025 11:52:05    May 20 2025 12:45:46
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_3DVarAOWCDA_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    May 20 2025 11:52:07    May 20 2025 12:05:59
202103250000        Done    May 20 2025 11:52:07    May 20 2025 15:16:01
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_hybAOWCDA_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    May 20 2025 11:52:09    May 20 2025 12:05:40
202103250000        Done    May 20 2025 11:52:09    May 20 2025 14:45:43
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SWA_gefs_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    May 20 2025 11:52:11    May 20 2025 12:55:55
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SW_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    May 20 2025 11:52:13    May 20 2025 12:45:53
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_atm3DVar_extended_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    May 20 2025 11:52:15    May 20 2025 12:06:08
202112210000        Done    May 20 2025 11:52:15    May 20 2025 16:25:58
202112210600        Done    May 20 2025 11:52:15    May 20 2025 17:11:05
202112211200        Done    May 20 2025 12:11:17    May 20 2025 19:35:53
202112211800      Active    May 20 2025 16:31:17             -          
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmaerosnowDA_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    May 20 2025 11:52:17    May 20 2025 12:10:59
202112201800        Done    May 20 2025 11:52:17    May 20 2025 14:15:44
202112210000        Done    May 20 2025 11:52:17    May 20 2025 14:05:44
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmDA_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    May 20 2025 11:52:19    May 20 2025 12:06:12
202112210000        Done    May 20 2025 11:52:19    May 20 2025 14:11:20
202112210600        Done    May 20 2025 11:52:19    May 20 2025 13:51:15
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48mx500_S2SW_cyc_gfs_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    May 20 2025 11:52:21    May 20 2025 12:05:52
202112201800        Done    May 20 2025 11:52:21    May 20 2025 16:01:13
202112210000        Done    May 20 2025 11:52:21    May 20 2025 16:45:46
202112211800        Done    May 20 2025 12:11:01    May 20 2025 16:40:55
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_ufs_hybatmDA_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    May 20 2025 11:52:23    May 20 2025 12:06:16
202402240000        Done    May 20 2025 11:52:23    May 20 2025 14:41:21
202402240600        Done    May 20 2025 11:52:23    May 20 2025 14:26:09
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96mx100_S2S_pr3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
199405010000        Done    May 20 2025 11:52:25    May 20 2025 13:25:52
```

The one DEAD job is from case C96_atm3DVar_extended
```
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_atm3DVar_extended_pr3642
202112211800    gfs_gempak_f147-f168                   196633401                DEAD                 271         2        1731.0
```

A check of `gfs_gempak_f147-f168.log.0` finds that the job was killed for exceeding the specified 30 minute wall clock limit
```
Erro=>> PBS: job killed: walltime 1811 exceeded limit 1800
forrtl: error (78): process killed (SIGTERM)
forrtl: error (78): process killed (SIGTERM)
```
The rerun `gfs_gempak_f147-f168.log` contains the traceback
```
cat: OUTPUT.4009021: No such file or directory
++ JGFS_ATMOS_GEMPAK[1]postamble /lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/pr3642/jobs/JGFS_ATMOS_GEMPAK 1747774323 0
/lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/pr3642/ush/preamble.sh: line 63: 4026337 Terminated              err_chk
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source
gplt               0000000000496CEB  Unknown               Unknown  Unknown
libpthread-2.31.s  0000149FC49FD910  Unknown               Unknown  Unknown
libc-2.31.so       0000149FC40E50CA  clock_nanosleep       Unknown  Unknown
libc-2.31.so       0000149FC40EAE43  nanosleep             Unknown  Unknown
libc-2.31.so       0000149FC40EAD5A  sleep                 Unknown  Unknown
gplt               0000000000473EBE  Unknown               Unknown  Unknown
gplt               00000000004106EE  Unknown               Unknown  Unknown
gplt               0000000000405164  Unknown               Unknown  Unknown
gplt               00000000004E0F76  Unknown               Unknown  Unknown
libc-2.31.so       0000149FC403E24D  __libc_start_main     Unknown  Unknown
gplt               0000000000404D6A  Unknown               Unknown  Unknown
Error in message send = 22
itype, ichan, nwords,2,7307322,3
```
I'm not sure what the problem is.  What does `Error in message send = 22` mean?

The `gfs_gempak_f123-f144.log.0` also was terminated due to exceeding the specified 1800 second wall clock limit.
```
 GEMPAK-NAGRIB2>=>> PBS: job killed: walltime 1814 exceeded limit 1800
++ JGFS_ATMOS_GEMPAK[1]postamble /lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/pr3642/jobs/JGFS_ATMOS_GEMPAK 1747771667 0
/lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/pr3642/ush/preamble.sh: line 63: 963213 Terminated              "${SCRgfs}/exgfs_atmos_nawips.sh" "40km" "${FHR3}" "GFS_GEMPAK_WWB" "${COMOUT_ATMOS_GEMPAK_40km}"
```

The `gfs_gempak_f123-f144.log` rerun completed within the 1800 second walltime.

These failures do not appear to be related to the changes in this PR.

@DavidHuber-NOAA and @KateFriedman-NOAA :  Have you seen wall time failures in C96_atm3DVar_extended gfs_gempak jobs from other PRs?
author_association: CONTRIBUTOR
user: login: KateFriedman-NOAA
created_at: 2025-05-21T11:41:55Z
body: > Have you seen wall time failures in C96_atm3DVar_extended gfs_gempak jobs from other PRs?

@RussTreadon-NOAA Yes unfortunately, this is a known issue we see in almost every CI suite on WCOSS2 currently. See https://github.com/NOAA-EMC/global-workflow/issues/3630. You can ignore it.
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-21T15:23:32Z
body: @DavidNew-NOAA :  [`DavidNew-NOAA:feature/marine-analysis-jedi-class`](https://github.com/DavidNew-NOAA/global-workflow/tree/feature/marine-analysis-jedi-class) is [four commits](https://github.com/DavidNew-NOAA/global-workflow/compare/feature/marine-analysis-jedi-class...NOAA-EMC%3Aglobal-workflow%3Adevelop) behind the head of [`NOAA-EMC:develop`](https://github.com/NOAA-EMC/global-workflow/tree/develop).  These four commits represent 115 files changed.  Many of these changes are file deletion or renaming.

Shall we update `DavidNew-NOAA:feature/marine-analysis-jedi-class` or are you in the midst of debugging which would benefit from keeping the branch _as is_ until debugging is complete?
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-21T15:25:33Z
body: @RussTreadon-NOAA I'm in the midst of debugging, but I anticipate 1-2 hours of work max and then I will merge develop and push.
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-21T17:09:00Z
body: @RussTreadon-NOAA @CoryMartin-NOAA @guillaumevernieres OK, I made some revisions to this PR. 

1. I made a method in `marine_analysis.py` that aids in generating a YAML for the `soca_diag_stats` applications. I tested it, and it now generates the csv files that it should.
2. The `clean_empty_obsspaces` method in the `Jedi` class is now invoked by an optional boolean argument. This allows me to initialize the atm LETKF solver application YAML in the `initialize` method of `atmens_analysis.py` (as it was before).
3. If the `clean_empty_obsspaces` completely empties all obs spaces, then it will error. This way, we catch the problem and the user knows where/when/how it happened.

All this required GDASApp and jcb-gdas companion PRs:
https://github.com/NOAA-EMC/jcb-gdas/pull/126
https://github.com/NOAA-EMC/GDASApp/pull/1707

I tested these changes in the marine variational analysis jobs in my current test experiment, but I'd like to retest all four JEDI CI cases just to be safe.
author_association: CONTRIBUTOR
user: login: DavidNew-NOAA
created_at: 2025-05-22T17:17:15Z
body: @aerorahul @KateFriedman-NOAA I having successfully run `C96C48_ufs_hybatmDA`, `C96C48_hybatmaerosnowDA`, `C48mx500_3DVarAOWCDA`, and `C48mx500_hybAOWCDA` to completion on Hera with this branch. I think final CI will run successfully now on this branch 🤞
author_association: CONTRIBUTOR
user: login: emcbot
created_at: 2025-05-22T18:52:26Z
body: CI Tests set up to run in /lfs/h2/emc/ptmp/emc.global/PR/PR_3642/RUNTESTS on WCOSS
author_association: NONE
user: login: emcbot
created_at: 2025-05-22T19:31:24Z
body: Experiment C48_S2SWA_gefs **FAILED** on Gaeac6 in Build# 1 with error logs:
```
/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/COMROOT/C48_S2SWA_gefs_308f9070/logs/2021032312/gefs_atmos_prod_mem001_f006.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/COMROOT/C48_S2SWA_gefs_308f9070/logs/2021032312/gefs_atmos_prod_mem001_f012.log
```

Follow link here to view the contents of the above file(s): [(gefs_atmos_prod_mem001_f006.log)](https://gist.github.com/emcbot/4cd7d4efb7df2630dde264c02a801810) [(gefs_atmos_prod_mem001_f012.log)](https://gist.github.com/emcbot/60c8af790175e69dc2340590df5fb4d2)
author_association: NONE
user: login: emcbot
created_at: 2025-05-22T19:31:35Z
body: Experiment C48_S2SWA_gefs **FAILED** on Gaeac6 in Build# 1 in
`/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/EXPDIR/C48_S2SWA_gefs_308f9070`
author_association: NONE
user: login: emcbot
created_at: 2025-05-22T20:32:21Z
body: Experiment C48mx500_hybAOWCDA **FAILED** on Gaeac6 in Build# 1 with error logs:
```
/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/COMROOT/C48mx500_hybAOWCDA_308f9070/logs/2021032500/gdas_arch_tars.log
```

Follow link here to view the contents of the above file(s): [(gdas_arch_tars.log)](https://gist.github.com/emcbot/af33682dc4260fbffb25df13c2d78a42)
author_association: NONE
user: login: emcbot
created_at: 2025-05-22T20:32:29Z
body: Experiment C48mx500_hybAOWCDA **FAILED** on Gaeac6 in Build# 1 in
`/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/EXPDIR/C48mx500_hybAOWCDA_308f9070`
author_association: NONE
user: login: emcbot
created_at: 2025-05-22T20:56:58Z
body: Experiment C96C48mx500_S2SW_cyc_gfs **FAILED** on Gaeac6 in Build# 1 with error logs:
```
/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_308f9070/logs/2021122018/gdas_arch_tars.log
```

Follow link here to view the contents of the above file(s): [(gdas_arch_tars.log)](https://gist.github.com/emcbot/c88e0203e2c2c3a17779bf2ad5d7d875)
author_association: NONE
user: login: emcbot
created_at: 2025-05-22T20:57:06Z
body: Experiment C96C48mx500_S2SW_cyc_gfs **FAILED** on Gaeac6 in Build# 1 in
`/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/EXPDIR/C96C48mx500_S2SW_cyc_gfs_308f9070`
author_association: NONE
user: login: emcbot
created_at: 2025-05-22T21:27:26Z
body: Experiment C48mx500_3DVarAOWCDA **FAILED** on Gaeac6 in Build# 1 with error logs:
```
/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_308f9070/logs/2021032500/gdas_arch_tars.log
```

Follow link here to view the contents of the above file(s): [(gdas_arch_tars.log)](https://gist.github.com/emcbot/2462e30be44e29c980e08434d0f9ab4c)
author_association: NONE
user: login: emcbot
created_at: 2025-05-22T21:27:34Z
body: Experiment C48mx500_3DVarAOWCDA **FAILED** on Gaeac6 in Build# 1 in
`/gpfs/f6/drsa-precip3/world-shared/global/CI/3642/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_308f9070`
author_association: NONE
user: login: DavidNew-NOAA
created_at: 2025-05-22T22:04:09Z
body: I thought https://github.com/NOAA-EMC/GDASApp/pull/1707 was already merged, but it wasn't, so this branch is pointing to the wrong GDASApp develop without the necessary changes that allowed my local testing to succeed.
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-23T10:11:36Z
body: **WCOSS2 g-w CI**

`feature/marine-analysis-jedi-class` at 308f90707 installed on Cactus in `/lfs/h2/emc/ptmp/emc.global/PR/PR_3642/global-workflow`.  g-w CI run with the following results

```
/lfs/h2/emc/ptmp/emc.global/PR/PR_3642/RUNTESTS/EXPDIR/C48_ATM_3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    May 22 2025 18:52:28    May 22 2025 19:45:34
 
/lfs/h2/emc/ptmp/emc.global/PR/PR_3642/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    May 22 2025 18:52:31    May 22 2025 19:05:37
202103250000        Done    May 22 2025 18:52:31    May 22 2025 20:30:31
 
/lfs/h2/emc/ptmp/emc.global/PR/PR_3642/RUNTESTS/EXPDIR/C48mx500_hybAOWCDA_3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    May 22 2025 18:52:33    May 22 2025 19:05:41
202103250000        Done    May 22 2025 18:52:33    May 22 2025 20:10:29
 
/lfs/h2/emc/ptmp/emc.global/PR/PR_3642/RUNTESTS/EXPDIR/C48_S2SWA_gefs_3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    May 22 2025 18:52:39    May 22 2025 20:00:46
 
/lfs/h2/emc/ptmp/emc.global/PR/PR_3642/RUNTESTS/EXPDIR/C48_S2SW_extended_3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    May 22 2025 18:52:43    May 22 2025 21:15:58
202103231800        Done    May 22 2025 18:52:43    May 22 2025 21:15:58
 
/lfs/h2/emc/ptmp/emc.global/PR/PR_3642/RUNTESTS/EXPDIR/C48_S2SW_3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    May 22 2025 18:52:36    May 22 2025 19:55:50
 
/lfs/h2/emc/ptmp/emc.global/PR/PR_3642/RUNTESTS/EXPDIR/C96_atm3DVar_extended_3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    May 22 2025 18:52:46    May 22 2025 19:10:57
202112210000        Done    May 22 2025 18:52:46    May 22 2025 23:20:43
202112210600        Done    May 22 2025 18:52:46    May 23 2025 00:15:47
202112211200        Done    May 22 2025 19:16:03    May 23 2025 01:00:45
202112211800        Done    May 22 2025 23:25:56    May 23 2025 04:15:43
 
SKIP C96_atm3DVar on wcoss2
 
/lfs/h2/emc/ptmp/emc.global/PR/PR_3642/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    May 22 2025 18:52:49    May 22 2025 19:16:08
202112201800        Done    May 22 2025 18:52:49    May 22 2025 21:20:45
202112210000        Done    May 22 2025 18:52:49    May 22 2025 21:16:14
 
/lfs/h2/emc/ptmp/emc.global/PR/PR_3642/RUNTESTS/EXPDIR/C96C48_hybatmDA_3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    May 22 2025 18:52:51    May 22 2025 19:11:07
202112210000        Done    May 22 2025 18:52:51    May 22 2025 21:16:17
202112210600        Done    May 22 2025 18:52:51    May 22 2025 20:56:02
 
/lfs/h2/emc/ptmp/emc.global/PR/PR_3642/RUNTESTS/EXPDIR/C96C48mx500_S2SW_cyc_gfs_3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    May 22 2025 18:52:53    May 22 2025 19:11:11
202112201800        Done    May 22 2025 18:52:53    May 22 2025 21:20:51
202112210000        Done    May 22 2025 18:52:53    May 22 2025 22:20:58
202112211800        Done    May 22 2025 19:16:14    May 22 2025 22:16:01
 
/lfs/h2/emc/ptmp/emc.global/PR/PR_3642/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    May 22 2025 18:52:56    May 22 2025 19:11:16
202402240000        Done    May 22 2025 18:52:56    May 22 2025 21:46:04
202402240600        Done    May 22 2025 18:52:56    May 22 2025 21:31:42
 
/lfs/h2/emc/ptmp/emc.global/PR/PR_3642/RUNTESTS/EXPDIR/C96mx100_S2S_3642
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
199405010000        Done    May 22 2025 18:52:59    May 22 2025 20:31:19
```

All jobs successfully completed in each case run on Cactus.
author_association: CONTRIBUTOR