Repository: NOAA-EMC/GDASApp
Pull request #1714: Update JEDI hashes
Author: login: RussTreadon-NOAA
Created: 2025-05-24T09:50:41Z | Merged: 2025-05-27T20:22:00Z

# Description

This PR updates JEDI hashes in `sorc/`.

# Companion PRs

- [x] jcb-gdas PR [#98](https://github.com/NOAA-EMC/jcb-gdas/pull/98)

# Issues

Resolves [#1573](https://github.com/NOAA-EMC/GDASApp/issues/1573)

# Automated CI tests to run in Global Workflow

- [ ] atm_jjob <!-- JEDI atm single cycle DA !-->
- [ ] C96C48_ufs_hybatmDA <!-- JEDI atm cycled DA !-->
- [ ] C96C48_hybatmaerosnowDA  <!-- JEDI aero/snow cycled DA !-->
- [ ] C48mx500_3DVarAOWCDA <!-- JEDI low-res marine 3DVar cycled DA  !-->
- [ ] C48mx500_hybAOWCDA <!-- JEDI marine hybrid envar cycled DA !-->
- [ ] C96C48_hybatmDA <!-- GSI atm cycled DA !-->



Diff:
diff --git a/modulefiles/GDAS/ursa.intel.lua b/modulefiles/GDAS/ursa.intel.lua
index c9122c501..72b39377b 100644
--- a/modulefiles/GDAS/ursa.intel.lua
+++ b/modulefiles/GDAS/ursa.intel.lua
@@ -11,6 +11,7 @@ prepend_path("MODULEPATH", '/contrib/spack-stack/spack-stack-1.9.1/envs/ue-oneap
 load("stack-oneapi/2024.2.1")
 load("stack-intel-oneapi-mpi/2021.13")
 load("stack-python/3.11.7")
+load("intel-oneapi-mkl/2024.2.1")
 
 load("cmake/3.30.2")
 load("gettext/0.22.5")
diff --git a/parm/atm/jcb-base.yaml.j2 b/parm/atm/jcb-base.yaml.j2
index 612412031..bf02f7883 100644
--- a/parm/atm/jcb-base.yaml.j2
+++ b/parm/atm/jcb-base.yaml.j2
@@ -26,7 +26,7 @@ minimizer: DRPCG
 final_diagnostics_departures: anlmob
 final_prints_frequency: PT3H
 number_of_outer_loops: 2
-analysis_variables: [ua,va,t,ps,sphum,ice_wat,liq_wat,o3mr]
+analysis_variables: [eastward_wind,northward_wind,air_temperature,air_pressure_at_surface,water_vapor_mixing_ratio_wrt_moist_air,cloud_liquid_ice,cloud_liquid_water,ozone_mass_mixing_ratio]
 
 # Model things
 # ------------
@@ -118,7 +118,7 @@ bias_files:
 # -------------------------
 local_ensemble_da_solver: GETKF
 
-increment_variables: [ua,va,delz,delp,t,ps,sphum,ice_wat,liq_wat,o3mr]
+increment_variables: [eastward_wind,northward_wind,layer_thickness,air_pressure_thickness,air_temperature,air_pressure_at_surface,water_vapor_mixing_ratio_wrt_moist_air,cloud_liquid_ice,cloud_liquid_water,ozone_mass_mixing_ratio]
 
 # Veritcal localization for GETKF
 vl_fraction_of_retained_variance: 0.750
diff --git a/parm/jcb-gdas b/parm/jcb-gdas
index 50b09af2a..b6b1d022f 160000
--- a/parm/jcb-gdas
+++ b/parm/jcb-gdas
@@ -1 +1 @@
-Subproject commit 50b09af2aec0057e9f68b07c476a75c2796de9a3
+Subproject commit b6b1d022f0863398e41766c90c2441f24ea1e8b6
diff --git a/sorc/fv3-jedi b/sorc/fv3-jedi
index 598c5c316..80ee76686 160000
--- a/sorc/fv3-jedi
+++ b/sorc/fv3-jedi
@@ -1 +1 @@
-Subproject commit 598c5c31602aafca27e1b773ef2683a5fc453391
+Subproject commit 80ee7668636b546f76f4282f72e949006eb29205
diff --git a/sorc/ioda b/sorc/ioda
index 1b4484bf1..afbdf7138 160000
--- a/sorc/ioda
+++ b/sorc/ioda
@@ -1 +1 @@
-Subproject commit 1b4484bf17d942889cd36592d0e83fab3c8a8421
+Subproject commit afbdf71388b0266a0a87c0a8a3482a0f2349749e
diff --git a/sorc/iodaconv b/sorc/iodaconv
index a56eef6bd..5a445d498 160000
--- a/sorc/iodaconv
+++ b/sorc/iodaconv
@@ -1 +1 @@
-Subproject commit a56eef6bdca700b4356eb3b3608af642e880549d
+Subproject commit 5a445d4988c14e81be8dfc465441dbabbfcf009d
diff --git a/sorc/oops b/sorc/oops
index d7ea8de03..7af87d2e0 160000
--- a/sorc/oops
+++ b/sorc/oops
@@ -1 +1 @@
-Subproject commit d7ea8de039ee0a5a5e016b1b7afab6c6072c328a
+Subproject commit 7af87d2e074e4c9b6b86e3d15f40e9096f7e17b7
diff --git a/sorc/saber b/sorc/saber
index 8009aed05..67b1ceece 160000
--- a/sorc/saber
+++ b/sorc/saber
@@ -1 +1 @@
-Subproject commit 8009aed05836111174d62c4b4298be524efa343a
+Subproject commit 67b1ceeceed2c7b25e686e904b3c5eb43d1b31c0
diff --git a/sorc/soca b/sorc/soca
index e736ab273..756981657 160000
--- a/sorc/soca
+++ b/sorc/soca
@@ -1 +1 @@
-Subproject commit e736ab2732f2dc5512c0b50269eed5c3b3044b48
+Subproject commit 756981657425db438c07c042789b7d15f91dfd25
diff --git a/sorc/ufo b/sorc/ufo
index 7cbaae3e0..9a6c6bad6 160000
--- a/sorc/ufo
+++ b/sorc/ufo
@@ -1 +1 @@
-Subproject commit 7cbaae3e03624829992491b35cbe3a8652650885
+Subproject commit 9a6c6bad6ac9692c8f6e01df54718fa120225ed1
diff --git a/test/testreference/atm_jjob_lgetkf_observer.ref b/test/testreference/atm_jjob_lgetkf_observer.ref
index fee8c429b..15ea1b4e1 100644
--- a/test/testreference/atm_jjob_lgetkf_observer.ref
+++ b/test/testreference/atm_jjob_lgetkf_observer.ref
@@ -96,21 +96,21 @@ f10m                                         | Min:+9.2995810057091766e-01 Max:+
 ----------------------------------------------------------------------------------------------------
 H(x) for member 1:
 
-AMSUA N19 nobs= 83277 Min=201.8068522604288, Max=282.1977354171773, RMS=233.8559301511745
+AMSUA N19 nobs= 83277 Min=201.8068522615118, Max=282.197735417631, RMS=233.8559301511771
 
-sondes nobs= 2812 Min=-33.46923510113491, Max=237.4511515234925, RMS=66.49450460686083
+sondes nobs= 8481 Min=-33.72958083245494, Max=309.9582199847107, RMS=141.2113994268784
 
 H(x) for member 2:
 
-AMSUA N19 nobs= 83277 Min=201.6491448573896, Max=282.044303211175, RMS=233.8543791644236
+AMSUA N19 nobs= 83277 Min=201.6491448610736, Max=282.0443032103622, RMS=233.8543791644092
 
-sondes nobs= 2812 Min=-30.53144491152982, Max=237.5347783619015, RMS=66.47644691726953
+sondes nobs= 8481 Min=-30.5314449120472, Max=310.201637251467, RMS=141.1993510065985
 
 H(x) for member 3:
 
-AMSUA N19 nobs= 83277 Min=201.5822623085232, Max=282.1681017005528, RMS=233.8547828542548
+AMSUA N19 nobs= 83277 Min=201.5822623101754, Max=282.1681017018822, RMS=233.8547828542639
 
-sondes nobs= 2812 Min=-32.86095389873932, Max=237.9311435831063, RMS=66.51314984035037
+sondes nobs= 8481 Min=-32.86095387229424, Max=310.3775309270029, RMS=141.1778363649958
 
 H(x) ensemble background mean: 
 
diff --git a/test/testreference/atm_jjob_lgetkf_solver.ref b/test/testreference/atm_jjob_lgetkf_solver.ref
index 5a4e03929..cd95fcd49 100644
--- a/test/testreference/atm_jjob_lgetkf_solver.ref
+++ b/test/testreference/atm_jjob_lgetkf_solver.ref
@@ -96,33 +96,33 @@ f10m                                         | Min:+9.2995810057091766e-01 Max:+
 ----------------------------------------------------------------------------------------------------
 H(x) for member 1:
 
-AMSUA N19 nobs= 83277 Min=201.8068542480469, Max=282.1977233886719, RMS=233.8559301395111
+AMSUA N19 nobs= 83277 Min=201.8068542480469, Max=282.1977233886719, RMS=233.8559301397531
 
-sondes nobs= 2812 Min=-33.46923446655273, Max=237.4511566162109, RMS=66.49450464374321
+sondes nobs= 8481 Min=-33.72957992553711, Max=309.9582214355469, RMS=141.2113995313577
 
 H(x) for member 2:
 
-AMSUA N19 nobs= 83277 Min=201.6491394042969, Max=282.0443115234375, RMS=233.8543791556629
+AMSUA N19 nobs= 83277 Min=201.6491394042969, Max=282.0443115234375, RMS=233.8543791552713
 
-sondes nobs= 2812 Min=-30.53144454956055, Max=237.5347747802734, RMS=66.47644682705078
+sondes nobs= 8481 Min=-30.53144454956055, Max=310.2016296386719, RMS=141.1993510651212
 
 H(x) for member 3:
 
-AMSUA N19 nobs= 83277 Min=201.5822601318359, Max=282.1680908203125, RMS=233.854782829338
+AMSUA N19 nobs= 83277 Min=201.5822601318359, Max=282.1680908203125, RMS=233.8547828296696
 
-sondes nobs= 2812 Min=-32.86095428466797, Max=237.9311370849609, RMS=66.51314989418812
+sondes nobs= 8481 Min=-32.86095428466797, Max=310.3775329589844, RMS=141.1778363269655
 
 H(x) ensemble background mean: 
 
-AMSUA N19 nobs= 83277 Min=201.7046763102213, Max=282.1362915039062, RMS=233.8550114699153
+AMSUA N19 nobs= 83277 Min=201.7046763102213, Max=282.1362915039062, RMS=233.8550114699761
 
-sondes nobs= 2812 Min=-32.28721110026041, Max=237.6390228271484, RMS=66.49058679825113
+sondes nobs= 8481 Min=-32.28721110026041, Max=310.1791280110677, RMS=141.1946022869963
 
 background y - H(x): 
 
-AMSUA N19 nobs= 83254 Min=-40.12262471516925, Max=14.39438883463544, RMS=1.52021880811151
+AMSUA N19 nobs= 83254 Min=-40.12262471516925, Max=14.39438883463544, RMS=1.520218808199241
 
-sondes nobs= 2812 Min=-6.339819272359213, Max=6.252817789713541, RMS=2.077638363669303
+sondes nobs= 3950 Min=-16.79891141255696, Max=16.0742301940918, RMS=4.060200046174113
 
 Background mean :
 ----------------------------------------------------------------------------------------------------
@@ -159,21 +159,21 @@ f10m                                         | Min:+9.3258902621206641e-01 Max:+
 Analysis mean :
 ----------------------------------------------------------------------------------------------------
 State print | number of fields = 28 | cube sphere face size: C48
-eastward_wind                                | Min:-5.0315151012088307e+01 Max:+8.9876394093713031e+01 RMS:+1.5970695781224096e+01
-northward_wind                               | Min:-7.0617201268531801e+01 Max:+7.7433449726355860e+01 RMS:+7.7146404824447874e+00
-air_temperature                              | Min:+1.7652661360280035e+02 Max:+3.1531640662494840e+02 RMS:+2.4990917048185136e+02
-layer_thickness                              | Min:-2.8742236232070200e+03 Max:-1.5810236377788424e+01 RMS:+9.8306082003743279e+02
-air_pressure_thickness                       | Min:+6.0599999954215089e-01 Max:+1.7748162500155472e+03 RMS:+9.5672871271542260e+02
+eastward_wind                                | Min:-5.0315150936846862e+01 Max:+8.9876392887765178e+01 RMS:+1.5970695782090216e+01
+northward_wind                               | Min:-7.0617188949045897e+01 Max:+7.7433452901272716e+01 RMS:+7.7146404832116575e+00
+air_temperature                              | Min:+1.7652661360280035e+02 Max:+3.1531640650599002e+02 RMS:+2.4990917048290376e+02
+layer_thickness                              | Min:-2.8742236232070095e+03 Max:-1.5810236370767129e+01 RMS:+9.8306082003702807e+02
+air_pressure_thickness                       | Min:+6.0599999954215089e-01 Max:+1.7748162496329298e+03 RMS:+9.5672871269830057e+02
 air_pressure_at_surface                      | Min:+5.3309177524748637e+04 Max:+1.0391290732286175e+05 RMS:+9.8892856414055801e+04
-water_vapor_mixing_ratio_wrt_moist_air       | Min:+0.0000000000000000e+00 Max:+1.9548966528437035e-02 RMS:+4.8538262896707044e-03
-cloud_liquid_ice                             | Min:+0.0000000000000000e+00 Max:+4.2005729394313881e-04 RMS:+1.0456783110622460e-05
-cloud_liquid_water                           | Min:+0.0000000000000000e+00 Max:+1.9000854625769036e-03 RMS:+3.5025871953604539e-05
+water_vapor_mixing_ratio_wrt_moist_air       | Min:+0.0000000000000000e+00 Max:+1.9548968712815695e-02 RMS:+4.8538262909497819e-03
+cloud_liquid_ice                             | Min:+0.0000000000000000e+00 Max:+4.2005702080881761e-04 RMS:+1.0456783079969593e-05
+cloud_liquid_water                           | Min:+0.0000000000000000e+00 Max:+1.9000848666799023e-03 RMS:+3.5025871969556609e-05
 snow_water                                   | Min:+0.0000000000000000e+00 Max:+0.0000000000000000e+00 RMS:+0.0000000000000000e+00
 rain_water                                   | Min:+0.0000000000000000e+00 Max:+0.0000000000000000e+00 RMS:+0.0000000000000000e+00
 graupel                                      | Min:+0.0000000000000000e+00 Max:+0.0000000000000000e+00 RMS:+0.0000000000000000e+00
 cloud_ice_number_concentration               | Min:+0.0000000000000000e+00 Max:+0.0000000000000000e+00 RMS:+0.0000000000000000e+00
 rain_number_concentration                    | Min:+0.0000000000000000e+00 Max:+0.0000000000000000e+00 RMS:+0.0000000000000000e+00
-ozone_mass_mixing_ratio                      | Min:+0.0000000000000000e+00 Max:+1.7891447711240337e-05 RMS:+4.4960701985932252e-06
+ozone_mass_mixing_ratio                      | Min:+0.0000000000000000e+00 Max:+1.7891447709183598e-05 RMS:+4.4960701987584135e-06
 geopotential_height_at_surface               | Min:-2.7018394470214840e+01 Max:+5.2697460937500000e+03 RMS:+6.3032575784066080e+02
 slmsk                                        | Min:+0.0000000000000000e+00 Max:+2.0000000000000000e+00 RMS:+6.5853328538727862e-01
 sheleg                                       | Min:+0.0000000000000000e+00 Max:+3.2527521006769342e+02 RMS:+2.2874231579198337e+01


Reviews:
user: login: shlyaeva
body: Thank you for updating the hashes! I have a couple questions on the differences in the references.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-27T12:13:05Z
user: login: DavidNew-NOAA
body: Looks good. Thanks @RussTreadon-NOAA . I was concerned as well about the test references, but Anna has already commented on those.

I will note that since we will no longer use fields metadata overrides after this PR, the `fv3jedi_fv3inc` and `fv3jedi_correction_increment` applications in GDASApp should be updated. They use several different geometries solely for the purpose of accessing different fields metadata overrides, even though the underlying physical geometry is the same for all states and increments. See for example https://github.com/NOAA-EMC/GDASApp/blob/e2aa02f03ee8abc0942b0c8c5b03e76ce8a08179/utils/fv3jedi/fv3jedi_correction_increment.h#L72. This can and should be done is a separate PR though. I'll make this comment an issue. 
state: DISMISSED
author_association: COLLABORATOR
submitted_at: 2025-05-27T12:36:49Z
user: login: CoryMartin-NOAA
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-27T12:48:23Z
user: login: danholdaway
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-27T13:11:11Z
user: login: CoryMartin-NOAA
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-27T13:12:56Z
user: login: RussTreadon-NOAA
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-27T18:10:56Z
user: login: RussTreadon-NOAA
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-27T18:11:55Z
user: login: CoryMartin-NOAA
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-27T18:34:56Z
user: login: CoryMartin-NOAA
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-27T19:53:04Z
user: login: danholdaway
body: Let's merge and keep going. Thanks for documenting everything Russ.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-27T20:11:15Z

Review comments:
diff_hunk: @@ -96,33 +96,33 @@ f10m                                         | Min:+9.2995810057091766e-01 Max:+
 ----------------------------------------------------------------------------------------------------
 H(x) for member 1:
 
-AMSUA N19 nobs= 83277 Min=201.8068542480469, Max=282.1977233886719, RMS=233.8559301395111
+AMSUA N19 nobs= 83277 Min=201.8068542480469, Max=282.1977233886719, RMS=233.8559301397531
 
-sondes nobs= 2812 Min=-33.46923446655273, Max=237.4511566162109, RMS=66.49450464374321
+sondes nobs= 8481 Min=-33.72957992553711, Max=309.9582214355469, RMS=141.2113995313577
path: test/testreference/atm_jjob_lgetkf_solver.ref
user: login: shlyaeva
body: Interesting: this is both a big change in the number of obs, and also a big difference in max, what's the reason for this?
created_at: 2025-05-27T12:11:56Z
author_association: COLLABORATOR
diff_hunk: @@ -96,33 +96,33 @@ f10m                                         | Min:+9.2995810057091766e-01 Max:+
 ----------------------------------------------------------------------------------------------------
 H(x) for member 1:
 
-AMSUA N19 nobs= 83277 Min=201.8068542480469, Max=282.1977233886719, RMS=233.8559301395111
+AMSUA N19 nobs= 83277 Min=201.8068542480469, Max=282.1977233886719, RMS=233.8559301397531
 
-sondes nobs= 2812 Min=-33.46923446655273, Max=237.4511566162109, RMS=66.49450464374321
+sondes nobs= 8481 Min=-33.72957992553711, Max=309.9582214355469, RMS=141.2113995313577
 
 H(x) for member 2:
 
-AMSUA N19 nobs= 83277 Min=201.6491394042969, Max=282.0443115234375, RMS=233.8543791556629
+AMSUA N19 nobs= 83277 Min=201.6491394042969, Max=282.0443115234375, RMS=233.8543791552713
 
-sondes nobs= 2812 Min=-30.53144454956055, Max=237.5347747802734, RMS=66.47644682705078
+sondes nobs= 8481 Min=-30.53144454956055, Max=310.2016296386719, RMS=141.1993510651212
 
 H(x) for member 3:
 
-AMSUA N19 nobs= 83277 Min=201.5822601318359, Max=282.1680908203125, RMS=233.854782829338
+AMSUA N19 nobs= 83277 Min=201.5822601318359, Max=282.1680908203125, RMS=233.8547828296696
 
-sondes nobs= 2812 Min=-32.86095428466797, Max=237.9311370849609, RMS=66.51314989418812
+sondes nobs= 8481 Min=-32.86095428466797, Max=310.3775329589844, RMS=141.1778363269655
 
 H(x) ensemble background mean: 
 
-AMSUA N19 nobs= 83277 Min=201.7046763102213, Max=282.1362915039062, RMS=233.8550114699153
+AMSUA N19 nobs= 83277 Min=201.7046763102213, Max=282.1362915039062, RMS=233.8550114699761
 
-sondes nobs= 2812 Min=-32.28721110026041, Max=237.6390228271484, RMS=66.49058679825113
+sondes nobs= 8481 Min=-32.28721110026041, Max=310.1791280110677, RMS=141.1946022869963
 
 background y - H(x): 
 
-AMSUA N19 nobs= 83254 Min=-40.12262471516925, Max=14.39438883463544, RMS=1.52021880811151
+AMSUA N19 nobs= 83254 Min=-40.12262471516925, Max=14.39438883463544, RMS=1.520218808199241
 
-sondes nobs= 2812 Min=-6.339819272359213, Max=6.252817789713541, RMS=2.077638363669303
+sondes nobs= 3950 Min=-16.79891141255696, Max=16.0742301940918, RMS=4.060200046174113
path: test/testreference/atm_jjob_lgetkf_solver.ref
user: login: shlyaeva
body: Here the number of obs is significantly smaller than for H(x) (3950 vs 8481), do we understand why?
created_at: 2025-05-27T12:12:40Z
author_association: COLLABORATOR
diff_hunk: @@ -96,21 +96,21 @@ f10m                                         | Min:+9.2995810057091766e-01 Max:+
 ----------------------------------------------------------------------------------------------------
 H(x) for member 1:
 
-AMSUA N19 nobs= 83277 Min=201.8068522604288, Max=282.1977354171773, RMS=233.8559301511745
+AMSUA N19 nobs= 83277 Min=201.8068522615118, Max=282.197735417631, RMS=233.8559301511771
 
-sondes nobs= 2812 Min=-33.46923510113491, Max=237.4511515234925, RMS=66.49450460686083
+sondes nobs= 8481 Min=-33.72958083245494, Max=309.9582199847107, RMS=141.2113994268784
path: test/testreference/atm_jjob_lgetkf_observer.ref
user: login: CoryMartin-NOAA
body: Related to the concerns about changes in the references. This has to be departures, not H(x) right? How could we have a H(x) of temperature be -33K?
created_at: 2025-05-27T12:48:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -96,21 +96,21 @@ f10m                                         | Min:+9.2995810057091766e-01 Max:+
 ----------------------------------------------------------------------------------------------------
 H(x) for member 1:
 
-AMSUA N19 nobs= 83277 Min=201.8068522604288, Max=282.1977354171773, RMS=233.8559301511745
+AMSUA N19 nobs= 83277 Min=201.8068522615118, Max=282.197735417631, RMS=233.8559301511771
 
-sondes nobs= 2812 Min=-33.46923510113491, Max=237.4511515234925, RMS=66.49450460686083
+sondes nobs= 8481 Min=-33.72958083245494, Max=309.9582199847107, RMS=141.2113994268784
path: test/testreference/atm_jjob_lgetkf_observer.ref
user: login: danholdaway
body: Perhaps it includes all variables in the min/max/rms calculation. Because the max looks like a reasonable max for temperature while the min looks like a reasonable min for winds.
created_at: 2025-05-27T13:11:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -96,21 +96,21 @@ f10m                                         | Min:+9.2995810057091766e-01 Max:+
 ----------------------------------------------------------------------------------------------------
 H(x) for member 1:
 
-AMSUA N19 nobs= 83277 Min=201.8068522604288, Max=282.1977354171773, RMS=233.8559301511745
+AMSUA N19 nobs= 83277 Min=201.8068522615118, Max=282.197735417631, RMS=233.8559301511771
 
-sondes nobs= 2812 Min=-33.46923510113491, Max=237.4511515234925, RMS=66.49450460686083
+sondes nobs= 8481 Min=-33.72958083245494, Max=309.9582199847107, RMS=141.2113994268784
path: test/testreference/atm_jjob_lgetkf_observer.ref
user: login: CoryMartin-NOAA
body: Good point. I bet that's what it is. Makes these diagnostics super useful to include all variables! :-)
created_at: 2025-05-27T13:12:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -96,33 +96,33 @@ f10m                                         | Min:+9.2995810057091766e-01 Max:+
 ----------------------------------------------------------------------------------------------------
 H(x) for member 1:
 
-AMSUA N19 nobs= 83277 Min=201.8068542480469, Max=282.1977233886719, RMS=233.8559301395111
+AMSUA N19 nobs= 83277 Min=201.8068542480469, Max=282.1977233886719, RMS=233.8559301397531
 
-sondes nobs= 2812 Min=-33.46923446655273, Max=237.4511566162109, RMS=66.49450464374321
+sondes nobs= 8481 Min=-33.72957992553711, Max=309.9582214355469, RMS=141.2113995313577
path: test/testreference/atm_jjob_lgetkf_solver.ref
user: login: RussTreadon-NOAA
body: Increased count and different stats occurred when ioda PR https://github.com/JCSDA-internal/ioda/pull/1448 and oops PR https://github.com/JCSDA-internal/oops/pull/2870 were pulled in via stable-nightly update.
created_at: 2025-05-27T18:10:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -96,33 +96,33 @@ f10m                                         | Min:+9.2995810057091766e-01 Max:+
 ----------------------------------------------------------------------------------------------------
 H(x) for member 1:
 
-AMSUA N19 nobs= 83277 Min=201.8068542480469, Max=282.1977233886719, RMS=233.8559301395111
+AMSUA N19 nobs= 83277 Min=201.8068542480469, Max=282.1977233886719, RMS=233.8559301397531
 
-sondes nobs= 2812 Min=-33.46923446655273, Max=237.4511566162109, RMS=66.49450464374321
+sondes nobs= 8481 Min=-33.72957992553711, Max=309.9582214355469, RMS=141.2113995313577
 
 H(x) for member 2:
 
-AMSUA N19 nobs= 83277 Min=201.6491394042969, Max=282.0443115234375, RMS=233.8543791556629
+AMSUA N19 nobs= 83277 Min=201.6491394042969, Max=282.0443115234375, RMS=233.8543791552713
 
-sondes nobs= 2812 Min=-30.53144454956055, Max=237.5347747802734, RMS=66.47644682705078
+sondes nobs= 8481 Min=-30.53144454956055, Max=310.2016296386719, RMS=141.1993510651212
 
 H(x) for member 3:
 
-AMSUA N19 nobs= 83277 Min=201.5822601318359, Max=282.1680908203125, RMS=233.854782829338
+AMSUA N19 nobs= 83277 Min=201.5822601318359, Max=282.1680908203125, RMS=233.8547828296696
 
-sondes nobs= 2812 Min=-32.86095428466797, Max=237.9311370849609, RMS=66.51314989418812
+sondes nobs= 8481 Min=-32.86095428466797, Max=310.3775329589844, RMS=141.1778363269655
 
 H(x) ensemble background mean: 
 
-AMSUA N19 nobs= 83277 Min=201.7046763102213, Max=282.1362915039062, RMS=233.8550114699153
+AMSUA N19 nobs= 83277 Min=201.7046763102213, Max=282.1362915039062, RMS=233.8550114699761
 
-sondes nobs= 2812 Min=-32.28721110026041, Max=237.6390228271484, RMS=66.49058679825113
+sondes nobs= 8481 Min=-32.28721110026041, Max=310.1791280110677, RMS=141.1946022869963
 
 background y - H(x): 
 
-AMSUA N19 nobs= 83254 Min=-40.12262471516925, Max=14.39438883463544, RMS=1.52021880811151
+AMSUA N19 nobs= 83254 Min=-40.12262471516925, Max=14.39438883463544, RMS=1.520218808199241
 
-sondes nobs= 2812 Min=-6.339819272359213, Max=6.252817789713541, RMS=2.077638363669303
+sondes nobs= 3950 Min=-16.79891141255696, Max=16.0742301940918, RMS=4.060200046174113
path: test/testreference/atm_jjob_lgetkf_solver.ref
user: login: RussTreadon-NOAA
body: Changes in printed statistics occurred when ioda PR https://github.com/JCSDA-internal/ioda/pull/1448 and oops PR https://github.com/JCSDA-internal/oops/pull/2870 were pulled in via stable-nightly update.
created_at: 2025-05-27T18:11:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -96,33 +96,33 @@ f10m                                         | Min:+9.2995810057091766e-01 Max:+
 ----------------------------------------------------------------------------------------------------
 H(x) for member 1:
 
-AMSUA N19 nobs= 83277 Min=201.8068542480469, Max=282.1977233886719, RMS=233.8559301395111
+AMSUA N19 nobs= 83277 Min=201.8068542480469, Max=282.1977233886719, RMS=233.8559301397531
 
-sondes nobs= 2812 Min=-33.46923446655273, Max=237.4511566162109, RMS=66.49450464374321
+sondes nobs= 8481 Min=-33.72957992553711, Max=309.9582214355469, RMS=141.2113995313577
path: test/testreference/atm_jjob_lgetkf_solver.ref
user: login: CoryMartin-NOAA
body: I think the old max looks wrong, 310K is a much more reasonable max temperature value for sondes, 237K would mean there is nothing tropospheric. Is this a Tv vs Ts issue? I think I recall the sondes from prepBUFR tend to be Ts in the stratosphere and Tv in troposphere
created_at: 2025-05-27T18:34:56Z
author_association: CONTRIBUTOR

PR comments:
user: login: RussTreadon-NOAA
created_at: 2025-05-27T10:32:43Z
body: The changes in this PR have been tested via g-w CI cases on Cactus (see g-w (#3690](PR https://github.com/NOAA-EMC/global-workflow/pull/3690)), Hera, Hercules, and Orion with successful results. 

GDASApp stable-nightly also successfully runs with the changes in the PR.

```
Automated GW-GDASApp Testing Results:
Machine: hera

Start: Tue May 27 00:42:43 UTC 2025 on hfe05
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Tue May 27 01:27:37 UTC 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Tue May 27 02:27:11 UTC 2025
Tests: 100% tests passed, 0 tests failed out of 135
```

author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-27T10:40:36Z
body: **Note**

The changes in GDASApp PR [#1589](https://github.com/NOAA-EMC/GDASApp/pull/1589) have been rolled into `feature/stable-nightly`.  This was done at [22c4aed](https://github.com/NOAA-EMC/GDASApp/pull/1714/commits/22c4aedb96de158c645ef3991e58abeadae389bc).

Approval and merger of this PR into GDASApp `develop` will close GDASApp PR [#1589](https://github.com/NOAA-EMC/GDASApp/pull/1589).
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-27T10:44:25Z
body: @shlyaeva and @guillaumevernieres  :  This PR updates the `sorc/soca` hash to  [7569816](https://github.com/NOAA-EMC/soca/tree/756981657425db438c07c042789b7d15f91dfd25).   This is the [`dev/emc`](https://github.com/NOAA-EMC/soca/tree/dev/emc) soca hash in sync with the authoritative soca at [b9a89f4](https://github.com/NOAA-EMC/soca/commit/b9a89f4a2a238ca85bdf59ca9a95673f5af5e791).
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-27T10:45:45Z
body: @danholdaway :  This PR includes changes from GDASApp PR [#1589](https://github.com/NOAA-EMC/GDASApp/pull/1589).  Approval and merger of this PR into GDASApp `develop` will close PR [#1589](https://github.com/NOAA-EMC/GDASApp/pull/1589)
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-27T10:48:15Z
body: Once this PR is reviewed, approved, and merged, the `sorc/gdas.cd` hash in g-w PR [#3690](https://github.com/NOAA-EMC/global-workflow/pull/3690) will be updated and that PR marked _Ready for review_.
author_association: CONTRIBUTOR
user: login: danholdaway
created_at: 2025-05-27T13:28:28Z
body: Presumably the main point of interest in these reference changes should be the big fv3-jedi PR. Moving from the Geometry fieldmetadata YAML files to the explicit specification of the mapping between internal variable name and file variable name in the main YAML. I'm looking at the changes to [atmosphere_background_ensemble.yaml.j2](https://github.com/NOAA-EMC/GDASApp/pull/1714#diff-205ac1d57c33e5ce36f6ea8a3b054b165311d13b1f192d13bb1a5ff695e87274) in the jcb PR and everything looks OK. @RussTreadon-NOAA can you point us to the sbatch logs from the observer run? Then if we can compare to the same log from the same run but from develop we could see where things diverge. Perhaps we will see differences in the prints from the State after it's read. It could be that either develop or this code has a bug somewhere in mapping an internal variable to a variable name in the file.
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-27T13:31:37Z
body: @danholdaway , I am currently running tests to identify the JEDI hash(es) responsible for the change in the reference counts.  The change in the reference values is **not** related to changes in GDASApp PR [#1589](https://github.com/NOAA-EMC/GDASApp/pull/1589)
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-27T18:00:22Z
body: Compile `feature/stable-nightly` with various JEDI hashes.  Going back to 

- [ioda @ 16acce9](https://github.com/jcsda/ioda/tree/16acce9ceff2c7d92dd76b4cf519640759332c4c)
- [oops @ d7ea8de](https://github.com/jcsda/oops/tree/d7ea8de039ee0a5a5e016b1b7afab6c6072c328a)
- [ufo @ 8c44fa3](https://github.com/jcsda/ufo/tree/8c44fa3edec46c68ed137a398db3df3821d3a8a4)

returns the `H(x) for member ` sonde counts back to `2812`.

**Test 1**
Revert ufo back to [ufo @ 9a6c6ba](https://github.com/jcsda/ufo/tree/9a6c6bad6ac9692c8f6e01df54718fa120225ed1).  This is the ufo hash in the current head of `feature/stable-nightly`.   Rerun `test_gdasapp_atm_jjob_ens_letkf`.  The sonde `H(x) for member` count remained `2812`.  

**Test 2**
Revert oops back to [oops @ 7af87d2](https://github.com/jcsda/oops/tree/7af87d2e074e4c9b6b86e3d15f40e9096f7e17b7).  This is the oops hash in the current head of `feature/stable-nightly`.   GDASApp compilation failed.   Set oops back to [oops @ d7ea8de](https://github.com/jcsda/oops/tree/d7ea8de039ee0a5a5e016b1b7afab6c6072c328a)

**Test 3**
Update [ioda hash](https://github.com/JCSDA/ioda/commits/develop) to [ioda @ 7f0c992](https://github.com/jcsda/ioda/tree/7f0c992c05464a6d91265379d4395b1f6b3244a5).  This is the ioda hash immediately after [ioda @ 16acce9](https://github.com/jcsda/ioda/tree/16acce9ceff2c7d92dd76b4cf519640759332c4c).  GDASApp compilation fails.  Set ioda hash back to  [ioda @ 16acce9](https://github.com/jcsda/ioda/tree/16acce9ceff2c7d92dd76b4cf519640759332c4c)

**Test 4**
Update [oops hash](https://github.com/JCSDA/oops/commits/develop) to [oops @ 7d819a4](https://github.com/JCSDA/oops/tree/7d819a41fd8cff66eace8ae04c33583801ed5a49).  This is the oops hash immediately after [oops @ d7ea8de](https://github.com/jcsda/oops/tree/d7ea8de039ee0a5a5e016b1b7afab6c6072c328a).  GDASApp compilation fails.

**Test 5**
Update ioda hash to [ioda @ 7f0c992](https://github.com/jcsda/ioda/tree/7f0c992c05464a6d91265379d4395b1f6b3244a5).  Update oops hash to [oops @ 7d819a4](https://github.com/JCSDA/oops/tree/7d819a41fd8cff66eace8ae04c33583801ed5a49).  These hashes correspond to ioda PR [#1448](https://github.com/JCSDA-internal/ioda/pull/1448) and oops PR [#2870](https://github.com/JCSDA-internal/oops/pull/2870).  GDASpp compiles.  Run `test_gdasapp_atm_jjob_ens_letkf`.  The sonde `H(x) for member` count changes to `8481`.  


@shlyaeva , @danholdaway , @CoryMartin-NOAA , @DavidNew-NOAA 
ioda PR  [#1448](https://github.com/JCSDA-internal/ioda/pull/1448) and oops PR [#2870](https://github.com/JCSDA-internal/oops/pull/2870) are changing the way observations are counted / screened in `test_gdasapp_atm_jjob_ens_letkf`.  Is this expected?  
  
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-27T18:32:24Z
body: Note the following in the `test_gdasapp_atm_jjob_ens_obs` log file

```
0: sondes processed vars: 6 Variables: airTemperature, specificHumidity, stationPressure, virtualTemperature, windEastward, windNorthward
0: sondes assimilated vars: 3 Variables: windEastward, windNorthward, airTemperature
```
While six sonde observation types are read only three types are assimilated:  u, v, and T.

Later on we see that 2827 observations of each type are read by the code.
```
0: Observer::finalize QC =
0: sondes airTemperature nlocs = 2827, nobs = 2827, min = 0, max = 19, avg = 9
0: sondes specificHumidity nlocs = 2827, nobs = 2827, min = 29, max = 29, avg = 29
0: sondes stationPressure nlocs = 2827, nobs = 2827, min = 29, max = 29, avg = 29
0: sondes virtualTemperature nlocs = 2827, nobs = 2827, min = 29, max = 29, avg = 29
0: sondes windEastward nlocs = 2827, nobs = 2827, min = 0, max = 19, avg = 7
0: sondes windNorthward nlocs = 2827, nobs = 2827, min = 0, max = 19, avg = 7
````

When ensemble member statistics are printed, we find `nobs= 8481.  Note that 3 x 2827 = 8481.

`0: Test     : H(x) for member ` for member 1, 2, and 3 each print `nobs= 8481` for sondes.   

Has the `H(x) for member` printout changed to represent the total number of observations of the given type (here, sondes) assimilated?

author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-27T18:45:21Z
body: Whether the old JEDI hashes (e.g., current `develop`) or new hashes (`feature/stable-nightly`), are the `H(x) from member` statistics printed over all observations in the source (e.g., `AMSUA N19` or `sondes` in this case)?

old hashes (`develop`)
```
0: Test     : H(x) for member 1:
0:
0: Test     : AMSUA N19 nobs= 83277 Min=201.8068522604288, Max=282.1977354171773, RMS=233.8559301511745
0:
0: Test     : sondes nobs= 2812 Min=-33.46923510113491, Max=237.4511515234925, RMS=66.49450460686083
```

new hashes (`feature/stable-nightly`)
```
0: Test     : H(x) for member 1:
0:
0: Test     : AMSUA N19 nobs= 83277 Min=201.8068522615118, Max=282.197735417631, RMS=233.8559301511771
0:
0: Test     : sondes nobs= 8481 Min=-33.72958083245494, Max=309.9582199847107, RMS=141.2113994268784
```

If 8481 observations represents 2827 u obs plus 2827 v obs plus 2827 temperature obs, then a min to max range from -33 to 309 is not unreasonable.  We can have a minimum u or v wind component -33 m/s.  A maximum temperature of 309 K is possible..
author_association: CONTRIBUTOR
user: login: CoryMartin-NOAA
created_at: 2025-05-27T18:50:03Z
body: I agree @RussTreadon-NOAA I am more curious as to why the old max value was 237, as that is too high to be a max wind (I think, that's like 737 cruising speed?) but too low to be a max temperature.
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-27T18:55:58Z
body: > I agree @RussTreadon-NOAA I am more curious as to why the old max value was 237, as that is too high to be a max wind (I think, that's like 737 cruising speed?) but too low to be a max temperature.

Not sure what / how the code was counting with the old hashes.  If indeed we are now lumping various observation types in a given obs space together and printing stats over the entire obs space, this is confusing.  It would be more meaningful to compute and print statistics as a function of observation type.
author_association: CONTRIBUTOR
user: login: shlyaeva
created_at: 2025-05-27T18:58:38Z
body: Interesting that fv3-jedi test references didn't need updating with those PRs. @RussTreadon-NOAA is there a still a run/com directory I can take a look at for these tests?
author_association: COLLABORATOR
user: login: RussTreadon-NOAA
created_at: 2025-05-27T19:00:44Z
body: The run directory is Hera `/scratch1/NCEPDEV/da/Russ.Treadon/CI/GDASApp/stable/20250527/global-workflow/sorc/gdas.cd/build/gdas/test/atm/global-workflow/testrun/RUNDIRS/gdas_test/enkfgdasatmensanl_18`.   

Log files are in `/scratch1/NCEPDEV/da/Russ.Treadon/CI/GDASApp/stable/20250527/global-workflow/sorc/gdas.cd/build/gdas/test/atm/global-workflow/testrun/`
author_association: CONTRIBUTOR
user: login: shlyaeva
created_at: 2025-05-27T19:01:20Z
body: > If indeed we are now lumping various observation types in a given obs space together and printing stats over the entire obs space, this is confusing. It would be more meaningful to compute and print statistics as a function of observation type.

Yes, that's how the code is working now, though it should be easy to change to print by variable.

Regardless of that, it's concerning that the number of obs is significantly different, I wonder if the masking isn't applied correctly or something like that. 
Did the solver result change at all?
author_association: COLLABORATOR
user: login: RussTreadon-NOAA
created_at: 2025-05-27T19:03:10Z
body: Yes, reference file `test/testreference/atm_jjob_lgetkf_solver.ref` is updated.
author_association: CONTRIBUTOR
user: login: shlyaeva
created_at: 2025-05-27T19:32:11Z
body: OK; I'm going to look into the LETKF issue in detail. Since LETKF is not on a critical path, should we merge this, and aim to resolve or explain the letkf differences asap after? (I am interested in seeing some soca bugfixes make it into gdasapp develop so gfsv17 experiments can start picking that up).
author_association: COLLABORATOR
user: login: CoryMartin-NOAA
created_at: 2025-05-27T19:33:13Z
body: I'm fine with that @shlyaeva if others are, given the need for GFSv17
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-27T19:50:50Z
body: GDASApp issue #1717 has been opened to further investigate the need to update the atmospheric LETKF observer and solver reference files.  

I agree with merging this PR into GDASApp `develop`.  This is required to move g-w PR [#3690](https://github.com/NOAA-EMC/global-workflow/pull/3690) forward.
author_association: CONTRIBUTOR
user: login: RussTreadon-NOAA
created_at: 2025-05-27T20:21:09Z
body: Thank you @shlyaeva , @CoryMartin-NOAA , @DavidNew-NOAA , and @danholdaway for your input.  I'll merge this PR into GDASApp `develop` and update g-w PR [#3690](https://github.com/NOAA-EMC/global-workflow/pull/3690).
author_association: CONTRIBUTOR