Repository: E3SM-Project/E3SM
Pull request #7350: EAMxx: fix horiz remap bug in case of "sparse" output locations
Author: login: bartgol
Labels: name: BFB
description: PR leaves answers BFB
name: EAMxx
description: C++ based E3SM atmosphere model (aka SCREAM)
Created: 2025-05-14T16:16:32Z | Merged: 2025-05-22T21:46:41Z

Fixes an issue in coarsening remap in case not all the fine grid cols participate to the remap. This bug affected ARM site output.

Fixes #7346 

[BFB]

---

I also changed HommeGridsManager, so that it generates grids with 0-based GIDs rather than 1-based. This is not too important for IO (as the IO layer is already subtracting the grid min GID when computing PIO decomp offsets), but it was making the horiz remapping a bit more brittle.

I will push a unit test for "sampling"-style coarsening remapping soon.

Linked issue #7346: EAMxx: online horizontal remapping appears to shift column index
Thanks to @hassanbeydoun for bringing this to my attention.

The issue arises when using online horizontal interpolation to map onto a subset of columns.  Hassan observed the behavior in a test case using `ne256pg2` and mapping onto the ARM sites using the remap file from the Cess-Potter simulations.  

Using the remap file: `$DIN_LOC_ROOT/atm/scream/maps/20230925_ARM_site_for_Cess_ne256_map.nc` the output data appears to be shifted in the `lat/lon` values.  Quick analysis confirms that the output data has data that matches the source grid column indices shifted by 248010.  In other words if column index 1 was meant to be part of the output instead column index 248011 is output.  Hassan's run and output can be found here: `/pscratch/sd/b/beydoun/e3sm_scratch/pm-gpu/ne256pg2_ne256pg2.F2010-SCREAMv1.Apr30.cess.rainfrac1.sepcldfracs.control/run`

I was able to reproduce this behavior with a `ne30pg2` simulation using the `$DIN_LOC_ROOT/atm/scream/maps/20230925_ARM_site_for_Cess_ne30_map.nc` remap file.  In this case the column indices appear to be shifted by 3382.  See `/pscratch/sd/a/adonahue/e3sm_scratch/pm-gpu/ne30pg2_ne30pg2.F2010-SCREAMv1.May07.2025_master` and `/pscratch/sd/a/adonahue/e3sm_scratch/pm-gpu/ne30pg2_ne30pg2.F2010-SCREAMv1.May07.2025_master.4`. 

The latter case used double the nodes to test if the number of ranks impacted the column shift.  It did not.
Linked issue comments: user: login: AaronDonahue
created_at: 2025-05-12T17:47:52Z
body: I'll add, the Cess-Potter results do show this shift in the column index.  In other words, they have the correct locations.  So this is a phenomenon that is either a) related to this particular run configuration or b) due to a code change that was integrated after the Cess-Potter run branch.
author_association: CONTRIBUTOR
user: login: bartgol
created_at: 2025-05-12T19:49:02Z
body: Have you tried running this on a CPU machine?
author_association: CONTRIBUTOR
user: login: AaronDonahue
created_at: 2025-05-12T20:11:05Z
body: > Have you tried running this on a CPU machine?

No, that is a good point.  I'll try that out

author_association: CONTRIBUTOR
user: login: AaronDonahue
created_at: 2025-05-12T22:50:15Z
body: @bartgol , just checked.  `pm-cpu` gives the same results.
author_association: CONTRIBUTOR
user: login: AaronDonahue
created_at: 2025-05-12T23:52:51Z
body: Okay, did some print statements and basically tracked down the cause:
```diff
diff --git a/components/eamxx/src/share/grid/remap/horiz_interp_remapper_data.cpp b/components/eamxx/src/share/grid/remap/horiz_interp_remapper_data.cpp
index 6b6d382b2f..d4a8e28c67 100644
--- a/components/eamxx/src/share/grid/remap/horiz_interp_remapper_data.cpp
+++ b/components/eamxx/src/share/grid/remap/horiz_interp_remapper_data.cpp
@@ -103,6 +103,7 @@ get_my_triplets (const std::string& map_file) const
 
   gid_type row_offset = global_map_file_min_row;
   gid_type col_offset = global_map_file_min_col;
+  printf("ASD: col and row offset = %d,\t%d\n",col_offset,row_offset);
   if (type==InterpType::Refine) {
     row_offset -= fine_grid->get_global_min_dof_gid();
   } else {
```
gives
```
1: ASD: col and row offset = 3383,      1
2: ASD: col and row offset = 3383,      1
0: ASD: col and row offset = 3383,      1
3: ASD: col and row offset = 3383,      1
```
in the output.  So `col_offset` , which is the global_map_file_min_col must be the cause.  

Which actually makes sense, the col_offset is supposed to always be 1.  Because the remap files use 1-based.  But because this is mapping ARM stations we are not using column 1 in the source mesh, like we do for typical horizontal remapping.
author_association: CONTRIBUTOR
user: login: AaronDonahue
created_at: 2025-05-12T23:57:25Z
body: I'm actually confused how this ever worked.  But the Cess-Potter data does work.  @bartgol do you recall if we made any changes that could have changed this?
author_association: CONTRIBUTOR
user: login: bartgol
created_at: 2025-05-13T03:09:29Z
body: Good detective work! That makes sense. And honestly, I am not sure what we got so far with ARM map files can be considered reliable. Imho, unless we thoroughly verify that the outputs are correct, they should be considered trash.

As for how to fix this, there are two options:
- a) change the map files, so that the row ids are in the range 1-Nsites
- b) forget about computing the offset, and ALWAYS assume that the map files GIDs are 1-based (as it is currently the case)

Imho, modifying the ARM map files may be easier.
author_association: CONTRIBUTOR
user: login: jsbamboo
created_at: 2025-05-13T04:11:52Z
body: @bartgol the issue seems to come from the "col" field in a non-global-src -> a dst mapping file, since "col" are gridcells in the src grid and "row" are gridcells in the dst grid (e.g., the "row" in `/global/cfs/cdirs/e3sm/inputdata/atm/scream/maps/20230925_ARM_site_for_Cess_ne256_map.nc` already starts from 1). As long as a mapping file is not covering the globe for the src grid, the "col" index typically does not start from 1

For ref, I checked an earlier wprrmxx output using a hires region -> 1x1 mapping file and saw the same problem as reported here. Following what found here, after changing from:
[def] `map_file_min_col = std::min(cols[id],map_file_min_col);` to 
[rev] `map_file_min_col = 1;` 
the outputs became consistent and matched as expected

* here shows the commands to generate this hires region -> 1x1 mapping file using nco. It's for a non-global-src -> dst, similar as the ARM map file:
```
ncremap -g 1x1_160E170E2.5N12.5N_SCRIP_ncremap.nc -G latlon=1,1#wesn=160,170,2.5,12.5
ncremap -a nco -s WP10ne32x32v1pg2_scrip.nc -g 1x1_160E170E2.5N12.5N_SCRIP_ncremap.nc  -m map_WP10ne32x32v1pg2_to_1x1_160E170E2.5N12.5N.nco.20250406.nc
```
and the resulting "col" does not start from 1, while "row" does (always starts from 1, regardless of the tool used?):
```
netcdf map_WP10ne32x32v1pg2_to_1x1_160E170E2.5N12.5N.nco.20250406 {
dimensions:
        n_s = 51884 ;
        n_a = 96240 ;
        n_b = 1 ;
 col = 36825, 36826, 36827, 36828, 36829, 36830, 36831, 36832, 36841, 36842, 
...
 row = 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 
...
```

* results comparison:
```
LC paths for file1 & file2: /p/lustre1/zhang73/E3SM_simulations/wprrmxx_p3/WPRRMxx_cosp_Nend_p3.WP10ne32x32v1pg2_WP10ne32x32v1pg2.F2010-SCREAMv1.dane/tests/1120x1_nhoursx1_UVTQ3h-s20141001-finicold-cosp3rad3tendx2-O5all-debug/run/
LC paths for file1 & file3: /p/lustre1/zhang73/E3SM_simulations/wprrmxx_p3/WPRRMxx_cospNend_coloffset_p3.WP10ne32x32v1pg2_WP10ne32x32v1pg2.F2010-SCREAMv1.dane/tests/1120x1_nhoursx1_UVTQ3h-s20141001-finicold-cosp3rad3tendx2-O5all-coloffset/run/
file 1: the manually regridded file | eamxx_wprrm.1hI.site_pg2_debug.h.INSTANT.nhours_x1.2014-10-01-00000.manual_ncremap_1x1.nc | 100918.9, 100865.8
file 2: the online output [def]     | eamxx_wprrm.1hI.1x1_160E170E2.5N12.5N_debug.h.INSTANT.nhours_x1.2014-10-01-00000.nc       | 99899.07, 99871.66
file 3: the online output [rev]     | eamxx_wprrm.1hI.1x1_160E170E2.5N12.5N_debug.h.INSTANT.nhours_x1.2014-10-01-00000.nc       | 100918.9, 100865.8
```

hope this provides some additional contex
author_association: CONTRIBUTOR
user: login: jsbamboo
created_at: 2025-05-13T05:40:51Z
body: The changes seem to be involved in this PR (https://github.com/E3SM-Project/scream/pull/2576/files). 

please note that in the older code (components/eamxx/src/share/grid/remap/coarsening_remapper.cpp), ~~the variable col_offset was likely misnamed (semantically, it should have been row_offset), but~~ (no, the semantic seems to be correct, it's col_offset calculated using row and applied to col)  the logic itself seems to be correct. The offset was consistently computed based _only_ on min(rows[id]), which corresponds to `map_file_min_row` in the updated code (components/eamxx/src/share/grid/remap/horiz_interp_remapper_data.cpp). There was no attempt to compute a separate `map_file_min_col`—effectively treating the column offset as always ~~zero~~ -> one—which is appropriate, since the col entries represent global src grid ids that can be sparse and do not require local reindexing (but just shift 1). The new map_file_min_col introduces potential issues when the mapping file is regional/sparse/masked remap files

would it be safer to only adjust the rows and leave the cols untouched or assume an offset of zero?
author_association: CONTRIBUTOR
user: login: bartgol
created_at: 2025-05-13T14:18:14Z
body: @jsbamboo you're right. The problem arises when either rows or cols don't span the whole index space, so for ARM sites map files, we'd still have the same problem. And yes, coarsening looks at cols ids when establishing which ones each rank needs, subtracting the global min col id, which will definitely be wrong.

At this point, I don't think modifying the map files becomes viable at all. While ~setting rows ids to [1,nsites] was a possibility~ _row ids are already in [1,nsites]_, there is no way to make the cols ids in [1,nsites] as well, or else we lose the arm location. At this point, I am leaning for hard coding (perhaps leaving a configurable option) the fact that ids are 1-based. Since our map files are for the foreseeable future always going to come from ncremap, this assumption is reasonable.

I'll work on a fix.
author_association: CONTRIBUTOR
user: login: jsbamboo
created_at: 2025-05-13T14:48:36Z
body: @bartgol thanks for the consideration and clarifications - that makes sense!
author_association: CONTRIBUTOR
user: login: mahf708
created_at: 2025-05-13T15:38:46Z
body: > Since our map files are for the foreseeable future always going to come from ncremap, this assumption is reasonable.

@bartgol: that's not a valid assumption. The regional mapping files come from here: https://github.com/E3SM-Project/eamxx-scripts/blob/master/util/create_regional_remapper.py.

IMHO: I think the overarching design (of letting users bring their own mapping files) is pretty wrong to begin with. It's like stretching the definition of remapping to the point of being nonsensical. What we should support is a limited set of functionality such that the user can request stuff like following list below, which should be impl'ed both stream-wise (like the functionality of mapping files) and variable-wise (which may prove a headache):

- output a stream/variable within `[ min_lat, max_lat, min_lon, max_lat ]` box
- output a stream/variable within an `x` degree/mile/km radius of point `[ lat , lon ]`
author_association: CONTRIBUTOR
user: login: AaronDonahue
created_at: 2025-05-13T15:54:16Z
body: > The changes seem to be involved in this PR (https://github.com/E3SM-Project/scream/pull/2576/files).

Thank you for digging this up @jsbamboo .  So this explains why the Cess-Potter output is correct because they started before this commit went in (@crterai is that correct?).  Sadly, this means the decadal runs are likely shifted.

> At this point, I don't think modifying the map files becomes viable at all. While setting rows ids to [1,nsites] was a possibility row ids are already in [1,nsites], there is no way to make the cols ids in [1,nsites] as well, or else we lose the arm location. At this point, I am leaning for hard coding (perhaps leaving a configurable option) the fact that ids are 1-based. Since our map files are for the foreseeable future always going to come from ncremap, this assumption is reasonable.

@bartgol , yes, I think this is a reasonable assumption. I would vote for default offset of 1 which can be defined elsewhere.  For simplicity, maybe we can request that any non-default offset has to be defined in the remap file metadata using the key `col_offset` and `row_offset`.  Noting that we need to stick with 1-based because the far and away most common use would be from `ncremap` files which are 1-based.

>IMHO: I think the overarching design (of letting users bring their own mapping files) is pretty wrong to begin with. It's like stretching the definition of remapping to the point of being nonsensical. 

@mahf708 , I can see your point and it was something we had originally considered.  However, the typical use case is for horizontal coarsening (ne1024->ne30, etc.).  This regional remapping or site based remapping is simply leveraging that built in capability.  So no matter what, anyone who understands horizontal remapping will have access to this use.  Note, I would be in favor adding your suggestion of a lat/lon box and calculating the remap online.  That wouldn't be too difficult, and would be a preferable way to specify regional output.
author_association: CONTRIBUTOR
user: login: bartgol
created_at: 2025-05-13T15:54:19Z
body: I am not familiar with that script, but a quick glance to that file seems to suggests that it still creates the same format that ncremap does (row/col/S arrays of size n_s, with row/col being 1-based IDs). The correct statement should have been

"Since our map files are for the foreseeable future _are always going to have the same format as ncremap generated ones (with 1-based ID numbering)_, this assumption is reasonable."

author_association: CONTRIBUTOR
user: login: AaronDonahue
created_at: 2025-05-13T15:55:38Z
body: @bartgol , yes.  It was a quick script that @crterai and I put together to generate regional output remap files.  It is designed to follow the ncremap format.
author_association: CONTRIBUTOR
user: login: bartgol
created_at: 2025-05-13T15:57:03Z
body: Ok, so I have fixed the issue, but I have one question: do you care if the sites are scrambled in the output file? You can still recover which site is which by inspecting the lat/lon coords, but maybe you want them in the same order as in the map file?
author_association: CONTRIBUTOR
user: login: mahf708
created_at: 2025-05-13T15:58:13Z
body: You can keep the horizontal remapping intact, which is not really affected by this bug. What I am saying is, I don't think it is appropriate to leverage horizontal remapping to do limited-domain output. It should be handled separately/independently of horiz remapping
author_association: CONTRIBUTOR
user: login: AaronDonahue
created_at: 2025-05-13T16:04:45Z
body: > Ok, so I have fixed the issue, but I have one question: do you care if the sites are scrambled in the output file? You can still recover which site is which by inspecting the lat/lon coords, but maybe you want them in the same order as in the map file?

I think it is important we maintain the expected "column" index in the output, which is defined by the `row` variable in the remap file.  

- For true horizontal remapping it would be very confusing if the output had shifted indices.  Most post-processing tools don't check lat/lon coordinates, they assume the index matches the grid it represents (i.e. ne30pg2 grid).
- For site based output, we could mine the lat/lon but it would be cumbersome.  The utility Naser pointed to generates a Site Index variable which allows users to quickly figure out which outputs correspond to which "stations" to make post-processing easy.

How difficult is to make sure the global "column" index of the output data matches the `row` index in the remap file.
author_association: CONTRIBUTOR
user: login: jsbamboo
created_at: 2025-05-13T16:14:38Z
body: @mahf708 I feel I somewhat understand your concern now. It sounds like, in your view, only a globe horizontal regrid counts as a "real" or "semantically correct" remapping, and thus outputs like site-based or sparse-grid are not truly remapping. from my perspective, as long as the mathematical core is computing the sparse matrix [S, col, row]—it's a form of remapping

speaking as a regular user, I personally would be happy to retain as much flexibility as possible (of course, under reasonable safeguards). For example, a common application is that downstream analysts working with k-scale simulation outputs may only care about a small watershed or an area within a city shapefile boundary. In these cases, allowing users to prepare and apply their own mapping files would be very convenient, while approaches like site output or box output would still require additional post-processing—of course, I very much like your and @AaronDonahue 's comments that adding classic eam-style output requests would indeed be very helpful! does this sound reasonable..? In this context, seems that the solution proposed by @bartgol  and @AaronDonahue  is still necessary
author_association: CONTRIBUTOR
user: login: mahf708
created_at: 2025-05-13T16:28:20Z
body: @jsbamboo, Yep, the bug should be fixed and the functionality should remain for power users --- I am not against both. For regular users though, I think a streamlined version of this is needed ... like I suggested above
author_association: CONTRIBUTOR
user: login: jsbamboo
created_at: 2025-05-13T16:34:14Z
body: @mahf708 thanks for walking through this - this all makes sense to me!
author_association: CONTRIBUTOR
user: login: bartgol
created_at: 2025-05-13T16:34:47Z
body: I think the users should be free to provide a remapping file that is arbitrary, provided that it follows the ncremap format. I don't think we should require that ALL model columns participate to some of the output columns. A _very aggressive_ coarsening may very well have some fine columns that are not used to compute the value of any coarse column. If one of those columns happen to have global ID 1 (in 1-based numbering),t he bug will manifest.

In other words, this bug _is_ affecting horizontal remapping, since we are assuming that the smallest row/col ids in the map file are the same thing as the smallest ids in the tgt/src grid, which is not necessarily the case for aggressive coarsening.

As for other localized output extraction, horiz remapping was the fastest way to achieve it (in terms of implementation), so it was the most reasonable choice at the time. It would be nice, of course, to implement some more targeted remapping functionalities, to allow users to request localized output directly from yaml file without having to produce a map file. For ARM sites, for instance, it may be faster, since it is a completely rank-local operation, and all the MPI calls can be avoided. For lat/lon box output, we can also avoid all comm, provided that one is only requesting the model cols in that box, and not, e.g., a regridded output to nlat x nlon points in such box.

Notice that in eamxx "remapping" does not necessarily mean "do something like ncremap", but rather "go from one layout to a different one", which may just involve relabeling/subviewing indices (like in the SE->PhysGLL remap), simply rearranging dofs (like in what twin-columns load balancing would do), or interpolating.
author_association: CONTRIBUTOR
user: login: bartgol
created_at: 2025-05-13T16:36:44Z
body: > > Ok, so I have fixed the issue, but I have one question: do you care if the sites are scrambled in the output file? You can still recover which site is which by inspecting the lat/lon coords, but maybe you want them in the same order as in the map file?
> 
> I think it is important we maintain the expected "column" index in the output, which is defined by the `row` variable in the remap file.
> 
> * For true horizontal remapping it would be very confusing if the output had shifted indices.  Most post-processing tools don't check lat/lon coordinates, they assume the index matches the grid it represents (i.e. ne30pg2 grid).
> * For site based output, we could mine the lat/lon but it would be cumbersome.  The utility Naser pointed to generates a Site Index variable which allows users to quickly figure out which outputs correspond to which "stations" to make post-processing easy.
> 
> How difficult is to make sure the global "column" index of the output data matches the `row` index in the remap file.

That makes sense. I was just asking in case the ARM output postprocessing was already doing a match to sites via lat/lon coords, in which case the order would be irrelevant. Tbh, I don't understand why I get the sites mixed up in the output, so I suspect something in the coarse grid construction is still amiss. When I'm done with this, the outputs should be in the correct order.
author_association: CONTRIBUTOR
user: login: mahf708
created_at: 2025-05-13T17:26:19Z
body: > As for other localized output extraction, horiz remapping was the fastest way to achieve it (in terms of implementation), so it was the most reasonable choice at the time. It would be nice, of course, to implement some more targeted remapping functionalities, to allow users to request localized output directly from yaml file without having to produce a map file. For ARM sites, for instance, it may be faster, since it is a completely rank-local operation, and all the MPI calls can be avoided. For lat/lon box output, we can also avoid all comm, provided that one is only requesting the model cols in that box, and not, e.g., a regridded output to nlat x nlon points in such box.

Exactly my point. "Fastest way" to achieve something is usually not the best way... A productive way to achieve this is to punt to diagnostics, but then we need the EAMxx IO library to support outputting only a select subset of columns. Not sure if this is readily available, but it shouldn't be too hard to achieve... or maybe it is hard to achieve, idk
author_association: CONTRIBUTOR
user: login: AaronDonahue
created_at: 2025-05-13T17:45:02Z
body: I would add, if we eventually create a diagnostic to handle this or make it a runtime option in the output.yaml file we will need to think carefully about how that will look.  For lat/lon box I think this is straightforward, just provide the bounds.  But for something like the ARM stations, we might have a list of O(10-20) stations, each with a lat/lon center point and desired radius around it.  As Naser pointed out a few posts back, this could be tricky to define from the user perspective.
author_association: CONTRIBUTOR
user: login: mahf708
created_at: 2025-05-13T17:48:07Z
body: Also, please add proper, realistic tests!!!!!! Obviously our testing isn't catching super critical bugs that waste tons of hours (a substantial part of production runs is now simply junk, no other way to put it)
author_association: CONTRIBUTOR
user: login: AaronDonahue
created_at: 2025-05-13T17:56:43Z
body: Yes, it is clear we didn't have a test in place to test "regional" remapping.  The bugfix should include one of those.  It is simple enough to generate one on the fly for the test or to create one that can be used with one of our "standalone" ne2pg2 tests.  This would have been caught with a baseline diff if we had it.
author_association: CONTRIBUTOR
user: login: mahf708
created_at: 2025-05-16T03:50:58Z
body: > Not sure if this is readily available, but it shouldn't be too hard to achieve... or maybe it is hard to achieve, idk

I thought about this a little more. Consider the following file out of an actual EAMxx run:

```
netcdf \3hi.INSTANT.nhours_x3.2020-08-29-10800 {
dimensions:
        time = UNLIMITED ; // (8 currently)
        dim10 = 10 ;
        dim8 = 8 ;
        ncol = 25165824 ;
        ilev = 129 ;
        lev = 128 ;
variables:
        double time(time) ;
                time:units = "days since 2019-08-01 00:00:00" ;
                time:long_name = "time" ;
                time:calendar = "noleap" ;
        float LiqNumberPath(time, ncol) ;
                LiqNumberPath:units = "kg/(kg m2)" ;
                LiqNumberPath:long_name = "LiqNumberPath" ;
                LiqNumberPath:_FillValue = 3.402824e+33f ;
```

First, note the arbitrary dim10, dim8 stuff. Pin that for a moment.

What I think we should do is, impl a generic diagnostics class that can take any arbitrary field (we have LiqNumberPath here, so I will use it) and then manipulate its ncol dimension as follows:


```
netcdf \3hi.INSTANT.nhours_x3.2020-08-29-10800 {
dimensions:
        time = UNLIMITED ; // (8 currently)
        dim10 = 10 ;
        dim8 = 8 ;
        ncol1 = 1;
        ncol25 = 25;
        ncol = 25165824 ;
        ilev = 129 ;
        lev = 128 ;
variables:
        double time(time) ;
                time:units = "days since 2019-08-01 00:00:00" ;
                time:long_name = "time" ;
                time:calendar = "noleap" ;
        float LiqNumberPath(time, ncol) ;
                LiqNumberPath:units = "kg/(kg m2)" ;
                LiqNumberPath:long_name = "LiqNumberPath" ;
                LiqNumberPath:_FillValue = 3.402824e+33f ;
        float LiqNumberPath_within_1km_of_lon50_lat60(time, ncol1) ;
                LiqNumberPath:units = "kg/(kg m2)" ;
                LiqNumberPath:long_name = "LiqNumberPath with 1 column" ;
                LiqNumberPath:_FillValue = 3.402824e+33f ;
        float LiqNumberPath_within_12km_of_lon50_lat60(time, ncol25) ;
                LiqNumberPath:units = "kg/(kg m2)" ;
                LiqNumberPath:long_name = "LiqNumberPath with 25 columns" ;
                LiqNumberPath:_FillValue = 3.402824e+33f ;
```

where ncol1 and ncol25 are stuff like the column nearest an ARM site and the 25 columns nearest an ARM site, respectively. For now, just imagine that we get those with something like radius LiqNumberPath_within_1km_of_lon50_lat60 and LiqNumberPath_within_12km_of_lon50_lat60. 

We should actually be able to pull this off in a relatively straightforward manner with little work. The two questions I have:

- is there way to guarantee the aribtrarily defined ncol1 and ncol25 follow the indexing of ncol or do we have to carefully manipulate that by carrying knowledge about global indexing of ncol around to inside the diag/proc impl? Otherwise, we may need to pass geo info with ncol1,ncol2 like lat1,lat25,lon1,lon25 as we already do for ncol...
- is the stuff in the inside of the file "too hard" for regular users to get quickly grasp?? If it is too hard, then it may defeat the point :(
author_association: CONTRIBUTOR
user: login: bartgol
created_at: 2025-05-16T14:59:38Z
body: > is there way to guarantee the aribtrarily defined ncol1 and ncol25 follow the indexing of ncol or do we have to carefully manipulate that by carrying knowledge about global indexing of ncol around to inside the diag/proc impl? Otherwise, we may need to pass geo info with ncol1,ncol2 like lat1,lat25,lon1,lon25 as we already do for ncol...

I'm not sure what you are asking. The indices in an nc dimension span [0,length). So for `LiqNumberPath_within_1km_of_lon50_lat60`, it makes no sense to ask for anything other than `LiqNumberPath_within_1km_of_lon50_lat60[0]` (once loaded in py or something). If you want, you can store a second (non time-dep) var `LiqNumberPath_within_1km_of_lon50_lat60_col_ids (ncol1)`, which tells you the gids of the closets col(s). Or you can save lat/lon, as you suggested. Again, I'm not sure if you are asking for having "original/model col info" in the nc file.

On a side note, we can probably open a discussion/issue to talk about new features, or changes in current nc output format/conventions...
author_association: CONTRIBUTOR
user: login: AaronDonahue
created_at: 2025-05-16T16:47:42Z
body: One issue that occurs to me, is the 25km from ARM site 1 may have a different dimension length than 25km from ARM site 2.  So having a generic `col25` dimension meant to represent "25 km from" might not work across the board.  We could have "25 nearest" or something, then it would always have length 25.
author_association: CONTRIBUTOR
user: login: mahf708
created_at: 2025-05-16T17:04:10Z
body: > One issue that occurs to me, is the 25km from ARM site 1 may have a different dimension length than 25km from ARM site 2. So having a generic `col25` dimension meant to represent "25 km from" might not work across the board. We could have "25 nearest" or something, then it would always have length 25.

that's right, the ncol25 is actually automatically detected and inserted.

> On a side note, we can probably open a discussion/issue to talk about new features, or changes in current nc output format/conventions...

yeah, will stop here for now, and will revisit later
author_association: CONTRIBUTOR

Diff:
diff --git a/components/eamxx/src/diagnostics/precip_surf_mass_flux.cpp b/components/eamxx/src/diagnostics/precip_surf_mass_flux.cpp
index 20c1d0668ca9..ebdbea9cf3c6 100644
--- a/components/eamxx/src/diagnostics/precip_surf_mass_flux.cpp
+++ b/components/eamxx/src/diagnostics/precip_surf_mass_flux.cpp
@@ -63,27 +63,29 @@ void PrecipSurfMassFlux::compute_diagnostic_impl()
   const auto use_liq = m_type & s_liq;
   const auto use_ice = m_type & s_ice;
 
-  std::int64_t dt;
+  double dt = 0;
   if (use_ice) {
     auto mass_ice = get_field_in("precip_ice_surf_mass");
     mass_ice_d = mass_ice.get_view<const Real*>();
 
     const auto& t_start = mass_ice.get_header().get_tracking().get_accum_start_time ();
     const auto& t_now   = mass_ice.get_header().get_tracking().get_time_stamp ();
-    dt = t_now-t_start;
-  }
-  if (use_liq) {
+    dt = t_now.seconds_from(t_start);
+    if (use_liq) {
+      // Ensure liq/ice have same accumulation times
+      auto mass_liq = get_field_in("precip_liq_surf_mass");
+      mass_liq_d = mass_liq.get_view<const Real*>();
+      EKAT_REQUIRE_MSG (t_now==mass_liq.get_header().get_tracking().get_time_stamp() and
+                        t_start==mass_liq.get_header().get_tracking().get_accum_start_time(),
+          "Error! Liquid and ice precip mass fields have different accumulation time stamps!\n");
+    }
+  } else if (use_liq) {
     auto mass_liq = get_field_in("precip_liq_surf_mass");
     mass_liq_d = mass_liq.get_view<const Real*>();
 
     const auto& t_start = mass_liq.get_header().get_tracking().get_accum_start_time ();
     const auto& t_now   = mass_liq.get_header().get_tracking().get_time_stamp ();
-    if (use_ice) {
-      EKAT_REQUIRE_MSG (dt==(t_now-t_start),
-          "Error! Liquid and ice precip mass fields have different accumulation time stamps!\n");
-    } else {
-      dt = t_now-t_start;
-    }
+    dt = t_now.seconds_from(t_start);
   }
 
   if (dt==0) {
diff --git a/components/eamxx/src/diagnostics/tests/precip_surf_mass_flux_tests.cpp b/components/eamxx/src/diagnostics/tests/precip_surf_mass_flux_tests.cpp
index c6dc2a2eab75..0062aad8036f 100644
--- a/components/eamxx/src/diagnostics/tests/precip_surf_mass_flux_tests.cpp
+++ b/components/eamxx/src/diagnostics/tests/precip_surf_mass_flux_tests.cpp
@@ -48,7 +48,7 @@ void run(std::mt19937_64& engine)
   auto gm = create_gm(comm,ncols);
 
   // Create timestep
-  const int dt=1800;
+  const double dt=1800;
 
   // Construct random input data
   using RPDF = std::uniform_real_distribution<Real>;
@@ -120,28 +120,29 @@ void run(std::mt19937_64& engine)
   diag_liq->compute_diagnostic();
   diag_ice->compute_diagnostic();
 
-  Field preicp_total_f = diag_total->get_diagnostic().clone();
-  Field preicp_liq_f   = diag_liq->get_diagnostic().clone();
-  Field preicp_ice_f   = diag_ice->get_diagnostic().clone();
-  preicp_total_f.deep_copy(0);
-  preicp_liq_f.deep_copy(0);
-  preicp_ice_f.deep_copy(0);
-  auto precip_total_v = preicp_total_f.get_view<Real*>();
-  auto precip_liq_v   = preicp_liq_f.get_view<Real*>();
-  auto precip_ice_v   = preicp_ice_f.get_view<Real*>();
+  Field precip_total_f = diag_total->get_diagnostic().clone();
+  Field precip_liq_f   = diag_liq->get_diagnostic().clone();
+  Field precip_ice_f   = diag_ice->get_diagnostic().clone();
+  precip_total_f.deep_copy(0);
+  precip_liq_f.deep_copy(0);
+  precip_ice_f.deep_copy(0);
+  auto precip_total_v = precip_total_f.get_view<Real*>();
+  auto precip_liq_v   = precip_liq_f.get_view<Real*>();
+  auto precip_ice_v   = precip_ice_f.get_view<Real*>();
   const auto rhodt = PC::RHO_H2O*dt;
   Kokkos::parallel_for("precip_total_surf_mass_flux_test",
                        typename KT::RangePolicy(0,ncols),
                        KOKKOS_LAMBDA(const int& icol) {
     precip_liq_v(icol)   = precip_liq_surf_mass_v(icol)/rhodt;
     precip_ice_v(icol)   = precip_ice_surf_mass_v(icol)/rhodt;
-    precip_total_v(icol) = precip_liq_v(icol) + precip_ice_v(icol);
+    precip_total_v(icol) = precip_ice_v(icol);
+    precip_total_v(icol) += precip_liq_surf_mass_v(icol)/rhodt;
   });
   Kokkos::fence();
 
-  REQUIRE(views_are_equal(diag_total->get_diagnostic(),preicp_total_f));
-  REQUIRE(views_are_equal(diag_liq->get_diagnostic(),preicp_liq_f));
-  REQUIRE(views_are_equal(diag_ice->get_diagnostic(),preicp_ice_f));
+  REQUIRE(views_are_equal(diag_total->get_diagnostic(),precip_total_f));
+  REQUIRE(views_are_equal(diag_liq->get_diagnostic(),precip_liq_f));
+  REQUIRE(views_are_equal(diag_ice->get_diagnostic(),precip_ice_f));
 
   // Finalize the diagnostic
   diag_total->finalize();
diff --git a/components/eamxx/src/diagnostics/tests/vertical_layer_tests.cpp b/components/eamxx/src/diagnostics/tests/vertical_layer_tests.cpp
index 56e9df9debd2..a33f9f2cddb0 100644
--- a/components/eamxx/src/diagnostics/tests/vertical_layer_tests.cpp
+++ b/components/eamxx/src/diagnostics/tests/vertical_layer_tests.cpp
@@ -134,10 +134,10 @@ void run (const std::string& diag_name, const std::string& location)
         // If interface, check value, otherwise perform int->mid averaging and check value
         auto int_val = prev_int_val + delta;
         if (location=="interfaces") {
-          REQUIRE_THAT(d_h(icol,ilev), Catch::Matchers::WithinRel(int_val,1e-5));
+          REQUIRE_THAT(d_h(icol,ilev), Catch::Matchers::WithinRel(int_val,Real(1e-5)));
         } else {
           auto mid_val = (int_val + prev_int_val) / 2;
-          REQUIRE_THAT(d_h(icol,ilev), Catch::Matchers::WithinRel(mid_val,1e-5));
+          REQUIRE_THAT(d_h(icol,ilev), Catch::Matchers::WithinRel(mid_val,Real(1e-5)));
         }
         prev_int_val = int_val;
       }
diff --git a/components/eamxx/src/dynamics/homme/interface/dyn_grid_mod.F90 b/components/eamxx/src/dynamics/homme/interface/dyn_grid_mod.F90
index 3460feab2fe2..26224fd21e4e 100644
--- a/components/eamxx/src/dynamics/homme/interface/dyn_grid_mod.F90
+++ b/components/eamxx/src/dynamics/homme/interface/dyn_grid_mod.F90
@@ -4,13 +4,11 @@ module dyn_grid_mod
   use shr_kind_mod,       only: r8 => shr_kind_r8
   use dimensions_mod,     only: nelem, nelemd, nelemdmax, np
   use edgetype_mod,       only: EdgeBuffer_t
+  use mpi
 
   implicit none
   private
 
-! We need MPI in here, so include it
-#include <mpif.h>
-
   public :: dyn_grid_init, get_my_dyn_data, cleanup_grid_init_data
 
   type (EdgeBuffer_t) :: edge
diff --git a/components/eamxx/src/dynamics/homme/interface/phys_grid_mod.F90 b/components/eamxx/src/dynamics/homme/interface/phys_grid_mod.F90
index 901cd18796ee..2f13cbc274d0 100644
--- a/components/eamxx/src/dynamics/homme/interface/phys_grid_mod.F90
+++ b/components/eamxx/src/dynamics/homme/interface/phys_grid_mod.F90
@@ -3,6 +3,7 @@ module phys_grid_mod
   use iso_c_binding, only: c_int, c_double
   use parallel_mod,  only: abortmp, MPIinteger_t, MPIreal_t
   use kinds,         only: iulog
+  use mpi
 
   implicit none
   private
@@ -55,8 +56,6 @@ module phys_grid_mod
 
   type(pg_specs_t), target :: pg_specs (pgN_min:pgN_max)
 
-! To get MPI_IN_PLACE and MPI_DATATYPE_NULL
-#include <mpif.h>
   ! Note: in this module, we often use MPI_IN_PLACE,0,MPI_DATATYPE_NULL
   !       for the src array specs in Allgatherv calls. These special values 
   !       inform MPI that src array is aliasing the dst one, so MPI will
diff --git a/components/eamxx/src/share/grid/remap/horiz_interp_remapper_data.cpp b/components/eamxx/src/share/grid/remap/horiz_interp_remapper_data.cpp
index 6b6d382b2fb9..2ae1c96df0c4 100644
--- a/components/eamxx/src/share/grid/remap/horiz_interp_remapper_data.cpp
+++ b/components/eamxx/src/share/grid/remap/horiz_interp_remapper_data.cpp
@@ -88,41 +88,10 @@ get_my_triplets (const std::string& map_file) const
 
   scorpio::release_file(map_file);
 
-  // 1.2 Dofs in grid are likely 0-based, while row/col ids in map file
-  // are likely 1-based. To match dofs, we need to offset the row/cols
-  // ids we just read in.
-  int map_file_min_row = std::numeric_limits<int>::max();
-  int map_file_min_col = std::numeric_limits<int>::max();
-  for (int id=0; id<nlweights; id++) {
-    map_file_min_row = std::min(rows[id],map_file_min_row);
-    map_file_min_col = std::min(cols[id],map_file_min_col);
-  }
-  int global_map_file_min_row, global_map_file_min_col;
-  comm.all_reduce(&map_file_min_row,&global_map_file_min_row,1,MPI_MIN);
-  comm.all_reduce(&map_file_min_col,&global_map_file_min_col,1,MPI_MIN);
-
-  gid_type row_offset = global_map_file_min_row;
-  gid_type col_offset = global_map_file_min_col;
-  if (type==InterpType::Refine) {
-    row_offset -= fine_grid->get_global_min_dof_gid();
-  } else {
-    col_offset -= fine_grid->get_global_min_dof_gid();
-  }
-  for (auto& id : rows) {
-    id -= row_offset;
-  }
-  for (auto& id : cols) {
-    id -= col_offset;
-  }
-
   // Create a grid based on the row gids I read in (may be duplicated across ranks)
-  std::vector<gid_type> unique_gids;
   const auto& gids = type==InterpType::Refine ? rows : cols;
-  for (auto gid : gids) {
-    if (not ekat::contains(unique_gids,gid)) {
-      unique_gids.push_back(gid);
-    }
-  }
+  std::set<gid_type> temp (gids.begin(),gids.end());
+  std::vector<gid_type> unique_gids (temp.begin(),temp.end());
   auto io_grid = std::make_shared<PointGrid> ("helper",unique_gids.size(),0,comm);
   auto io_grid_gids_h = io_grid->get_dofs_gids().get_view<gid_type*,Host>();
   int k = 0;
@@ -150,10 +119,18 @@ get_my_triplets (const std::string& map_file) const
   MPI_Type_create_struct (3,lengths,displacements,types,&mpi_triplet_t);
   MPI_Type_commit(&mpi_triplet_t);
 
-  // Create import-export
-  GridImportExport imp_exp (fine_grid,io_grid);
+  // Create import-export and gather my triplets
   std::map<int,std::vector<Triplet>> my_triplets_map;
-  imp_exp.gather(mpi_triplet_t,io_triplets,my_triplets_map);
+  try {
+    GridImportExport imp_exp (fine_grid,io_grid);
+    imp_exp.gather(mpi_triplet_t,io_triplets,my_triplets_map);
+  } catch (...) {
+    // Print the map file name, to help debugging
+    std::cout << "[HorizRemapperData] Something went wrong while performing a grid import-export operation.\n"
+              << " - map file name : " << map_file << "\n"
+              << " - fine grid name: " << fine_grid->name() << "\n";
+    throw;
+  }
   MPI_Type_free(&mpi_triplet_t);
 
   std::vector<Triplet> my_triplets;
diff --git a/components/eamxx/src/share/grid/remap/horiz_interp_remapper_data.hpp b/components/eamxx/src/share/grid/remap/horiz_interp_remapper_data.hpp
index e99a41e09bd0..760c3604701a 100644
--- a/components/eamxx/src/share/grid/remap/horiz_interp_remapper_data.hpp
+++ b/components/eamxx/src/share/grid/remap/horiz_interp_remapper_data.hpp
@@ -17,13 +17,16 @@ enum class InterpType {
   Coarsen
 };
 
-// A small struct to hold horiz remap data, which can
-// be shared across multiple horiz remappers
+// A small struct to hold horiz remap data, which can be shared across multiple horiz remappers
+// NOTE: the client will call the build method, which will read the map file, and create the
+//       CRS matrix data for online interpolation.
 struct HorizRemapperData {
   using KT = KokkosTypes<DefaultDevice>;
   template<typename T>
   using view_1d = typename KT::template view_1d<T>;
 
+  // The last argument specifies the base index for gids in the map file
+  // For ncremap-type files, all indices are 1-based
   void build (const std::string& map_file,
               const std::shared_ptr<const AbstractGrid>& fine_grid,
               const ekat::Comm& comm,
diff --git a/components/eamxx/src/share/io/tests/CMakeLists.txt b/components/eamxx/src/share/io/tests/CMakeLists.txt
index 4733c5f22312..90b5b3a589fb 100644
--- a/components/eamxx/src/share/io/tests/CMakeLists.txt
+++ b/components/eamxx/src/share/io/tests/CMakeLists.txt
@@ -88,6 +88,12 @@ CreateUnitTest(io_remap_test "io_remap_test.cpp"
   MPI_RANKS 1 ${SCREAM_TEST_MAX_RANKS}
 )
 
+## Test remap output when map file is sub-sampling (ARM-style)
+CreateUnitTest(io_horiz_sampling "io_horiz_sampling.cpp"
+  LIBS scream_io LABELS io remap
+  MPI_RANKS 1 ${SCREAM_TEST_MAX_RANKS}
+)
+
 ## Test single-column reader
 CreateUnitTest(io_scm_reader "io_scm_reader.cpp"
   LIBS scream_io LABELS io
diff --git a/components/eamxx/src/share/io/tests/io_horiz_sampling.cpp b/components/eamxx/src/share/io/tests/io_horiz_sampling.cpp
new file mode 100644
index 000000000000..62f6c306a045
--- /dev/null
+++ b/components/eamxx/src/share/io/tests/io_horiz_sampling.cpp
@@ -0,0 +1,165 @@
+#include <catch2/catch.hpp>
+#include <memory>
+
+#include "diagnostics/register_diagnostics.hpp"
+
+#include "share/io/eamxx_output_manager.hpp"
+#include "share/io/scorpio_input.hpp"
+#include "share/io/eamxx_scorpio_interface.hpp"
+#include "share/field/field_utils.hpp"
+#include "share/util/eamxx_setup_random_test.hpp"
+
+#include "share/grid/point_grid.hpp"
+
+namespace scream {
+
+Field create_f (const std::string& name,
+                const FieldLayout layout,
+                const std::string& grid_name)
+{
+  const auto nondim = ekat::units::Units::nondimensional();
+  FieldIdentifier fid(name,layout,nondim,grid_name);
+  Field f(fid);
+  f.allocate_view();
+  return f;
+}
+
+ekat::ParameterList output_params(const std::string& map_file)
+{
+  using strvec_t = std::vector<std::string>;
+
+  ekat::ParameterList params;
+  params.set<std::string>("filename_prefix","horiz_sampling");
+  params.set<std::string>("averaging_type","instant");
+  params.set<std::string>("floating_point_precision","real");
+  auto& oc = params.sublist("output_control");
+  oc.set<int>("frequency",1);
+  oc.set<std::string>("frequency_units","nsteps");
+  params.set<strvec_t>("field_names",{"s2d","s3d"});
+  params.set<std::string>("horiz_remap_file",map_file);
+
+  return params;
+}
+
+void print (const std::string& msg, const ekat::Comm& comm) {
+  if (comm.am_i_root()) {
+    printf("%s",msg.c_str());
+  }
+}
+
+TEST_CASE("io_remap_test","io_remap_test")
+{
+  using gid_type = AbstractGrid::gid_type;
+
+  // Init scorpio
+  ekat::Comm comm(MPI_COMM_WORLD);
+  scorpio::init_subsystem(comm);
+
+  util::TimeStamp t0 ({2000,1,1},{0,0,0});
+
+  // Random number generation
+  using RPDF  = std::uniform_real_distribution<Real>;
+  auto engine = setup_random_test(&comm);
+  RPDF pdf(0, 1);
+
+  // Create src grid
+  const std::string& gname = "point_grid";
+  const int ngcols_src = 10*comm.size();
+  const int nlevs = 4;
+  const auto src_grid = create_point_grid (gname,ngcols_src,nlevs,comm);
+  const int  nlcols_src = src_grid->get_num_local_dofs();
+  const auto gids_src_h = src_grid->get_dofs_gids().get_view<const gid_type*,Host>();
+
+  // Create remap file. The mapping strategy is simply to take every other column,
+  // but making sure to NOT pick the 1st one, so that the min col GID in map file is 2
+  print (" -> Create remap file ... \n",comm);
+  const int ngcols_tgt = ngcols_src / 2;
+  const int nlcols_tgt = nlcols_src / 2;
+  std::vector<int> col(nlcols_tgt), row(nlcols_tgt);
+  std::vector<Real> S(nlcols_tgt,1.0);
+  for (int i=0; i<nlcols_tgt; ++i) {
+    row[i] = i + nlcols_tgt*comm.rank();
+    col[i] = gids_src_h[2*i] + 1;
+  }
+
+  // Write remap data to file
+  const std::string remap_filename = "horiz_sampling_remap_np"+std::to_string(comm.size())+".nc";
+  scorpio::register_file(remap_filename, scorpio::FileMode::Write);
+
+  scorpio::define_dim(remap_filename,"n_a",ngcols_src);
+  scorpio::define_dim(remap_filename,"n_b",ngcols_tgt);
+  scorpio::define_dim(remap_filename,"n_s",ngcols_tgt);
+
+  scorpio::define_var(remap_filename,"col",   {"n_s"},"int");
+  scorpio::define_var(remap_filename,"row",   {"n_s"},"int");
+  scorpio::define_var(remap_filename,"S",     {"n_s"},"real");
+
+  // Linear decomposition
+  scorpio::set_dim_decomp(remap_filename,"n_s",comm.rank()*nlcols_tgt,nlcols_tgt);
+  scorpio::enddef(remap_filename);
+
+  scorpio::write_var(remap_filename,"row",   row.data());
+  scorpio::write_var(remap_filename,"col",   col.data());
+  scorpio::write_var(remap_filename,"S",     S.data());
+
+  scorpio::release_file(remap_filename);
+  print (" -> Create remap file ... done\n",comm);
+
+  // Create random source data
+  print (" -> Create source data ... \n",comm);
+
+  // Create the fields and randomize
+  auto s2d_src = create_f("s2d",src_grid->get_2d_scalar_layout(),gname);
+  auto s3d_src = create_f("s3d",src_grid->get_3d_scalar_layout(true),gname);
+  randomize(s2d_src,engine,pdf);
+  randomize(s3d_src,engine,pdf);
+
+  // Stuff fields in a FieldManager, since that's what OuputManager wants
+  auto fm = std::make_shared<FieldManager> (src_grid,RepoState::Closed);
+  fm->add_field(s2d_src);
+  fm->add_field(s3d_src);
+  fm->init_fields_time_stamp(t0);
+  print (" -> Create source data ... done\n",comm);
+
+  print (" -> Write output ... \n",comm);
+  double dt = 1.5;
+  OutputManager om;
+  auto params = output_params(remap_filename);
+  om.initialize (comm, params, t0, false);
+  om.setup(fm,{gname});
+
+  om.init_timestep(t0,dt);
+  om.run(t0+dt);
+  om.finalize();
+  print (" -> Write output ... done\n",comm);
+
+  print (" -> Check output ... \n",comm);
+
+  // Read output file
+  std::string filename = "horiz_sampling.INSTANT.nsteps_x1.np" + std::to_string(comm.size()) + "." + t0.to_string() + ".nc";
+  auto tgt_grid = create_point_grid(gname + "_tgt",ngcols_tgt,nlevs,comm);
+  auto s2d_tgt = create_f("s2d",tgt_grid->get_2d_scalar_layout(),gname+"_tgt");
+  auto s3d_tgt = create_f("s3d",tgt_grid->get_3d_scalar_layout(true),gname+"_tgt");
+
+  AtmosphereInput reader(filename,tgt_grid,{s2d_tgt,s3d_tgt});
+  reader.read_variables();
+  reader.finalize(); // manually finalize, or scorpio cleanup will complain about a file still open
+
+  // Check values
+  auto s2d_src_h = s2d_src.get_view<const Real* ,Host>();
+  auto s3d_src_h = s3d_src.get_view<const Real**,Host>();
+  auto s2d_tgt_h = s2d_tgt.get_view<const Real* ,Host>();
+  auto s3d_tgt_h = s3d_tgt.get_view<const Real**,Host>();
+  for (int i=0; i<nlcols_tgt; ++i) {
+    REQUIRE (s2d_tgt_h(i)==s2d_src_h(2*i+1));
+    for (int k=0; k<nlevs; ++k) {
+      REQUIRE (s3d_tgt_h(i,k)==s3d_src_h(2*i+1,k));
+    }
+  }
+  print (" -> Check output ... done\n",comm);
+  
+  // Cleanup scorpio
+  scorpio::finalize_subsystem();
+}
+
+} //namespace scream
diff --git a/components/eamxx/src/share/io/tests/io_remap_test.cpp b/components/eamxx/src/share/io/tests/io_remap_test.cpp
index 3f65f2966356..7deb0f817f53 100644
--- a/components/eamxx/src/share/io/tests/io_remap_test.cpp
+++ b/components/eamxx/src/share/io/tests/io_remap_test.cpp
@@ -87,10 +87,10 @@ TEST_CASE("io_remap_test","io_remap_test")
   const Real wgt = 0.4;
   for (int ii=0; ii<ncols_tgt_l; ii++) {
     const int src_col = 2*ii + ncols_src_l*io_comm.rank();
-    row.push_back(1+ii+ncols_tgt_l*io_comm.rank());
-    row.push_back(1+ii+ncols_tgt_l*io_comm.rank());
-    col.push_back(1+src_col);
-    col.push_back(1+src_col+1);
+    row.push_back(ii+ncols_tgt_l*io_comm.rank());
+    row.push_back(ii+ncols_tgt_l*io_comm.rank());
+    col.push_back(src_col);
+    col.push_back(src_col+1);
     S.push_back(wgt);
     S.push_back(1.0-wgt);
   }


Reviews:
user: login: AaronDonahue
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T16:47:31Z
user: login: AaronDonahue
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T16:50:40Z
user: login: bartgol
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T16:55:55Z
user: login: bartgol
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T16:56:52Z
user: login: AaronDonahue
body: This looks great, thanks!  We could also add a remapped output to one of our standalone tests and create an ARM style remap to apply to it.  Then we would catch any changes in DIFF to the baselines...
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T17:51:27Z
user: login: tcclevenger
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-22T20:26:02Z

Review comments:
diff_hunk: @@ -87,8 +85,9 @@ subroutine get_my_dyn_data (dg_gids, cg_gids, elgpgp, elgids, lat, lon)
       do ip=1,np
         do jp=1,np
           idof = (ie-1)*16+(jp-1)*4+ip
-          cg_gids(idof) = INT(el_cg_gids(ip,jp,ie),kind=c_int)
-          dg_gids(idof) = INT(el_dg_gids(ip,jp,ie),kind=c_int)
+          ! Fortran GIDs are 1-based, but we want 0-based ids
path: components/eamxx/src/dynamics/homme/interface/dyn_grid_mod.F90
user: login: AaronDonahue
body: Because this touches homme code, we will need to go the regular `next` -> `master` route.  Just noting here since it could be missed at integration.
created_at: 2025-05-14T16:47:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -17,17 +17,21 @@ enum class InterpType {
   Coarsen
 };
 
-// A small struct to hold horiz remap data, which can
-// be shared across multiple horiz remappers
+// A small struct to hold horiz remap data, which can be shared across multiple horiz remappers
+// NOTE: the client will call the build method, which will read the map file, and create the
+//       CRS matrix data for online interpolation.
 struct HorizRemapperData {
   using KT = KokkosTypes<DefaultDevice>;
   template<typename T>
   using view_1d = typename KT::template view_1d<T>;
 
+  // The last argument specifies the base index for gids in the map file
+  // For ncremap-type files, all indices are 1-based
   void build (const std::string& map_file,
               const std::shared_ptr<const AbstractGrid>& fine_grid,
               const ekat::Comm& comm,
-              const InterpType type);
+              const InterpType type,
+              const int map_file_base_gid = 1);
path: components/eamxx/src/share/grid/remap/horiz_interp_remapper_data.hpp
user: login: AaronDonahue
body: `map_file_base_gid` is default set to 1, I don't see where it could be set to a different value.  
created_at: 2025-05-14T16:50:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -87,8 +85,9 @@ subroutine get_my_dyn_data (dg_gids, cg_gids, elgpgp, elgids, lat, lon)
       do ip=1,np
         do jp=1,np
           idof = (ie-1)*16+(jp-1)*4+ip
-          cg_gids(idof) = INT(el_cg_gids(ip,jp,ie),kind=c_int)
-          dg_gids(idof) = INT(el_dg_gids(ip,jp,ie),kind=c_int)
+          ! Fortran GIDs are 1-based, but we want 0-based ids
path: components/eamxx/src/dynamics/homme/interface/dyn_grid_mod.F90
user: login: bartgol
body: This is not Homme, it's still eamxx. It's the _eamxx interface to homme_.
created_at: 2025-05-14T16:55:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -17,17 +17,21 @@ enum class InterpType {
   Coarsen
 };
 
-// A small struct to hold horiz remap data, which can
-// be shared across multiple horiz remappers
+// A small struct to hold horiz remap data, which can be shared across multiple horiz remappers
+// NOTE: the client will call the build method, which will read the map file, and create the
+//       CRS matrix data for online interpolation.
 struct HorizRemapperData {
   using KT = KokkosTypes<DefaultDevice>;
   template<typename T>
   using view_1d = typename KT::template view_1d<T>;
 
+  // The last argument specifies the base index for gids in the map file
+  // For ncremap-type files, all indices are 1-based
   void build (const std::string& map_file,
               const std::shared_ptr<const AbstractGrid>& fine_grid,
               const ekat::Comm& comm,
-              const InterpType type);
+              const InterpType type,
+              const int map_file_base_gid = 1);
path: components/eamxx/src/share/grid/remap/horiz_interp_remapper_data.hpp
user: login: bartgol
body: It's not. I added the hook in case we want to adapt later on. But maybe I'm just better off removing this?

Edit: I was planning to make scorpio_output parse the output yaml file to see if the user specified a base gid different from 1. But I see that this may be confusing, so I'll remove for now.
created_at: 2025-05-14T16:56:52Z
author_association: CONTRIBUTOR

PR comments:
user: login: bartgol
created_at: 2025-05-14T17:45:34Z
body: I added a unit test for sparse, sampling-like horiz remapping. To make sure it exercise the correct scenario, I also locally added the test in current master, and verified it fails.

This is good to go.
author_association: CONTRIBUTOR
user: login: bartgol
created_at: 2025-05-14T18:04:34Z
body: > This looks great, thanks! We could also add a remapped output to one of our standalone tests and create an ARM style remap to apply to it. Then we would catch any changes in DIFF to the baselines...

I think the unit test `io_horiz_sampling` should already cover that. But I'm not going to say no if someone wants to upgrade standalone (and CIME!) tests...
author_association: CONTRIBUTOR
user: login: bartgol
created_at: 2025-05-14T18:22:53Z
body: I am a bit concerned by the diffs in the CIME tests. I see cpl output diffing, which is very unexpected, so I need to dig a bit.
author_association: CONTRIBUTOR
user: login: AaronDonahue
created_at: 2025-05-14T19:55:23Z
body: Are the DIFFs related at all to #7322 going in?
author_association: CONTRIBUTOR
user: login: bartgol
created_at: 2025-05-14T20:03:14Z
body: > Are the DIFFs related at all to #7322 going in?

Can't be, since that only touches mam4xx compsets, and the diffing ones are the the basic eamxx and dpxx.
author_association: CONTRIBUTOR
user: login: mahf708
created_at: 2025-05-14T20:24:04Z
body: @bartgol would it be too much to ask to split the changes? Maybe start with the homme gridmanager separately?
author_association: CONTRIBUTOR
user: login: bartgol
created_at: 2025-05-14T22:01:17Z
body: Actually, I no longer remember why I needed that change in Homme. Reverting that commit makes everything pass. While I don't understand why that commit should cause the fail, considering that I no longer have a need for that change, I'll just remove it from the PR. I'm quickly testing on mappy one more time before pushing.

Edit: ok, tests seem to pass, let's see what CI says.
author_association: CONTRIBUTOR
user: login: mahf708
created_at: 2025-05-15T15:02:53Z
body: 

> > This looks great, thanks! We could also add a remapped output to one of our standalone tests and create an ARM style remap to apply to it. Then we would catch any changes in DIFF to the baselines...
> 
> I think the unit test `io_horiz_sampling` should already cover that. But I'm not going to say no if someone wants to upgrade standalone (and CIME!) tests...

It's really easy, you just need to upload a file to the input data dir with one ne4pg2 file that does some arm output... see:


https://github.com/E3SM-Project/E3SM/blob/dcc57a1507183817d44d5dfd66a6c83527855f07/components/eamxx/cime_config/testdefs/testmods_dirs/eamxx/prod/shell_commands#L45-L62


I really wish people would embrace the prod testmod as it covers TONS of the code and it replicates what we actually run... that test would've and should've caught both the cosp and this bug if someone helped fill the blanks... (I obviously added it to help get things started, remember I am not paid to do any dev... so I did for free essentially)
author_association: CONTRIBUTOR
user: login: bartgol
created_at: 2025-05-22T20:18:40Z
body: @AaronDonahue @tcclevenger tests are passing on this PR. Could you please take another look?
author_association: CONTRIBUTOR