Repository: ESMCI/cime

Issue:
ERI test failure with cime master branch
I'm not 100% sure this is a cime issue, but I suspect it is: In a version of CTSM where I have updated cime from cime5.6.10 to something close to master (bc8166bce023af36beeaaee64a3cd4187644f2eb), I am getting failures in ERI tests that produce daily history files. For example:

```
FAIL ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default COMPARE_base_hybrid
```

The failure comes from:

```
    File '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.clm2.h0.0003-01-09-00000.nc.hybrid' had no original counterpart in '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run' with suffix 'base'
```

This is from branch `update_cime` on https://github.com/billsacks/ctsm. Compared to the last working version of CTSM, the only differences are the cime update and a small change from @gold2718 or @ekluzek that changes the operation of pause-resume.

One possible suspect is the short-term archiver.

I'll look into this.


Discussion:
This does look like it's due to a cime change: With the new version of cime, the main ERI run directory contains these clm hist files:

```
ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.ref2.clm2.h0.0003-01-06-00000.nc.base
ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.clm2.h1.0003-01-09-00000.nc.base
```

whereas the previous version contained these:

```
ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g.clm2.h0.0003-01-09-00000.nc.base
ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g.clm2.h1.0003-01-09-00000.nc.base
```

Note the presence of 'ref2' in the new file name, which was not in the old one. I'm guessing this is due to some change in a regular expression match. Now I need to track down where.... Note that this (oddly) only affects the h0 file, not the h1 file.
More details on the differences in operation between the old (cime5.6.10) and new (bc8166b) versions of cime... no need to read through all this, I'm mainly recording what I've found to help organize my thoughts, and to help us confirm that we have fixed the issue later:

(1) Copying files to suffix 'base':

Old had:

```
2018-08-20 15:55:59: Copying hist files to suffix 'base'
  Copying hist files for model 'datm'
  Copying hist files for model 'clm'
    Copying '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540
c_g.clm2.h0.0003-01-09-00000.nc' to '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.
clm-default.GC.0820_1540c_g.clm2.h0.0003-01-09-00000.nc.base'
    Copying '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540
c_g.clm2.h1.0003-01-09-00000.nc' to '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.
clm-default.GC.0820_1540c_g.clm2.h1.0003-01-09-00000.nc.base'
```

New has:

```
2018-08-29 05:39:06: Copying hist files to suffix 'base'
  Copying hist files for model 'datm'
  Copying hist files for model 'clm'
    Copying '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.ref2.clm2.h0.0003-01-06-00000.nc' to '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.ref2.clm2.h0.0003-01-06-00000.nc.base'
    Copying '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.clm2.h1.0003-01-09-00000.nc' to '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.clm2.h1.0003-01-09-00000.nc.base'
```

Note the difference in the h0 file.

(2) Copying files to suffix rest:

Old had:

```
2018-08-20 15:56:17: Copying hist files to suffix 'rest'
  Copying hist files for model 'datm'
  Copying hist files for model 'clm'
    Copying '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540
c_g.clm2.h0.0003-01-09-00000.nc' to '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.
clm-default.GC.0820_1540c_g.clm2.h0.0003-01-09-00000.nc.rest'
    Copying '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540
c_g.clm2.h1.0003-01-09-00000.nc' to '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.
clm-default.GC.0820_1540c_g.clm2.h1.0003-01-09-00000.nc.rest'
```

New has:

```
2018-08-29 05:39:22: Copying hist files to suffix 'rest'
  Copying hist files for model 'datm'
  Copying hist files for model 'clm'
    Copying '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.ref2.clm2.h0.0003-01-06-00000.nc' to '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.ref2.clm2.h0.0003-01-06-00000.nc.rest'
    Copying '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.clm2.h1.0003-01-09-00000.nc' to '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.clm2.h1.0003-01-09-00000.nc.rest'
  Copying hist files for model 'sice'
  Copying hist files for model 'socn'
  Copying hist files for model 'mosart'
  Copying hist files for model 'sglc'
  Copying hist files for model 'swav'
  Copying hist files for model 'cpl'
    Copying '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.cpl.hi.0003-01-09-00000.nc' to '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.cpl.hi.0003-01-09-00000.nc.rest'
```

Again: Note the difference in the h0 file.

(3) Comparing hists: base vs. hybrid:

Old had:

```
2018-08-20 15:56:17: Comparing hists for case 'ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g' dir1='/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.
cheyenne_gnu.clm-default.GC.0820_1540c_g/run', suffix1='base',  dir2='/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run' suffix2='hybrid'
  comparing model 'datm'
    no hist files found for model datm
  comparing model 'clm'
    /glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g.clm2.h0.0003-01-09-00000.nc.base matched /glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g.clm2.h0.0003-01-09-00000.nc.hybrid
    /glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g.clm2.h1.0003-01-09-00000.nc.base matched /glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g.clm2.h1.0003-01-09-00000.nc.hybrid
```

New has:

```
2018-08-29 05:39:22: Comparing hists for case 'ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g' dir1='/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run', suffix1='base',  dir2='/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run' suffix2='hybrid'
  comparing model 'datm'
    no hist files found for model datm
  comparing model 'clm'
    File '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.clm2.h0.0003-01-09-00000.nc.hybrid' had no original counterpart in '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run' with suffix 'base'
    /glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.clm2.h1.0003-01-09-00000.nc.base matched /glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.clm2.h1.0003-01-09-00000.nc.hybrid
```

Note the "no original counterpart"

(4) Comparing hists: base vs. rest:

Old had:

```
2018-08-20 15:56:18: Comparing hists for case 'ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g' dir1='/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run', suffix1='base',  dir2='/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run' suffix2='rest'
  comparing model 'datm'
    no hist files found for model datm
  comparing model 'clm'
    /glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g.clm2.h0.0003-01-09-00000.nc.base matched /glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g.clm2.h0.0003-01-09-00000.nc.rest
    /glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g.clm2.h1.0003-01-09-00000.nc.base matched /glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g.clm2.h1.0003-01-09-00000.nc.rest
```

New has:

```
2018-08-29 05:39:22: Comparing hists for case 'ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g' dir1='/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run', suffix1='base',  dir2='/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run' suffix2='rest'
  comparing model 'datm'
    no hist files found for model datm
  comparing model 'clm'
    /glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.clm2.h1.0003-01-09-00000.nc.base matched /glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.clm2.h1.0003-01-09-00000.nc.rest
```

Note that the h0 file doesn't even show up in the new comparison.

(5) Baseline generation:

Old had:

```
  generating for model 'clm'
    generating baseline '/glade/p/cesmdata/cseg/cesm_baselines/ctsm1.0.dev009/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default/clm2.h0.0003-01-09-00000.nc' from file /glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g.clm2.h0.0003-01-09-00000.nc
    generating baseline '/glade/p/cesmdata/cseg/cesm_baselines/ctsm1.0.dev009/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default/clm2.h1.0003-01-09-00000.nc' from file /glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g.clm2.h1.0003-01-09-00000.nc
```

New has:

```
  generating for model 'clm'
    generating baseline '/glade/p/cesmdata/cseg/cesm_baselines/ctsm.update_cime.35d5a42e/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default/clm2.h0.0003-01-06-00000.nc' from file /glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.ref2.clm2.h0.0003-01-06-00000.nc
    generating baseline '/glade/p/cesmdata/cseg/cesm_baselines/ctsm.update_cime.35d5a42e/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default/clm2.h1.0003-01-09-00000.nc' from file /glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.clm2.h1.0003-01-09-00000.nc
```

Note wrong file used for the h0 file baseline generation.

(6) Baseline comparison:

Old had:

```
  comparing model 'clm'
    /glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g.clm2.h0.0003-01-09-00000.nc matched /glade/p/cesmdata/cseg/cesm_baselines/ctsm1.0.dev008/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default/clm2.h0.0003-01-09-00000.nc
    /glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0820_1540c_g.clm2.h1.0003-01-09-00000.nc matched /glade/p/cesmdata/cseg/cesm_baselines/ctsm1.0.dev008/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default/clm2.h1.0003-01-09-00000.nc
```

New has:

```
    File '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.ref2.clm2.h0.0003-01-06-00000.nc' had no compare counterpart in '/glade/p/cesmdata/cseg/cesm_baselines/ctsm1.0.dev009/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default' with suffix ''
    File '/glade/p/cesmdata/cseg/cesm_baselines/ctsm1.0.dev009/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default/clm2.h0.0003-01-09-00000.nc' had no original counterpart in '/glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run' with suffix ''
    /glade/scratch/sacks/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g/run/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default.GC.0829_0532c_g.clm2.h1.0003-01-09-00000.nc matched /glade/p/cesmdata/cseg/cesm_baselines/ctsm1.0.dev009/ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default/clm2.h1.0003-01-09-00000.nc
```

Note the two "no counterparts"


Most likely these different symptoms are all due to the same underlying problem.
Okay, I think I see the problem: It seems due to modifying a list when iterating over that list, in hist_utils.py:_get_all_hist_files.

From testing this code in isolation, this change seems to fix it; I'll test to see if this fixes the ERI case:

```diff
--- tester_old.py	2018-08-29 13:36:57.887331000 -0600
+++ tester2_new.py	2018-08-29 14:00:04.756923228 -0600
@@ -14,9 +14,7 @@
         test_hists.extend([os.path.join(from_dir,f) for f in os.listdir(from_dir) if pfile.search(f)])

     if ref_case:
-        for test in test_hists:
-            if ref_case in os.path.basename(test):
-                test_hists.remove(test)
+        test_hists = [h for h in test_hists if not (ref_case in os.path.basename(h))]

     test_hists = list(set(test_hists))
     test_hists.sort()
```
With this change in place, I reran `ERI_D_Ld9.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.clm-default` and it now passes the in-test comparisons and the baseline comparisons; I checked points 1, 2, 3, 4 and 6 from https://github.com/ESMCI/cime/issues/2790#issuecomment-417033877 and it seems like it is operating the same as in cime5.6.10 (I forgot to use the --generate flag, so didn't check point 5, but I assume that is fixed, too, since all of the other points are fixed).

I'll open a PR with this change.

Resolving patch:
diff --git a/scripts/lib/CIME/hist_utils.py b/scripts/lib/CIME/hist_utils.py
--- a/scripts/lib/CIME/hist_utils.py
+++ b/scripts/lib/CIME/hist_utils.py
@@ -44,9 +44,7 @@ def _get_all_hist_files(model, from_dir, file_extensions, suffix="", ref_case=No
         test_hists.extend([os.path.join(from_dir,f) for f in os.listdir(from_dir) if pfile.search(f)])
 
     if ref_case:
-        for test in test_hists:
-            if ref_case in os.path.basename(test):
-                test_hists.remove(test)
+        test_hists = [h for h in test_hists if not (ref_case in os.path.basename(h))]
 
     test_hists = list(set(test_hists))
     test_hists.sort()


Test patch: