Repository: ESMCI/cime

Issue:
compare_test_results namelist comparisons generate lots of FAILs
This is not a critical issue, because I don't really care about this functionality, but I wanted to report it because it could be confusing for others:

I ran compare_test_results like this:

```
Tools/compare_test_results -b clm4_5_16_r262.old -r tests_1026-1933c -t 1026-1933c_i | tee compare.out2
```

where the `clm4_5_16_r262.old` baseline directory was generated with the `--generate` flag to `create_test` (i.e., it was _not_ generated using `bless_test_results`).

This gave `NLCOMP` failures for a lot of tests, even though namelists didn't really change: ERS, ERI, NCK, and some CLM-specific tests (LII, LVG, SSP) all failed the namelist comparison.

From spot-checking a few, it looks like the problem relates to the timing of in-test namelist generation: When using the `--generate` flag, I think namelists are generated early in the test creation process, whereas this after-the-fact compare_test_results looks at namelists from the end of the test. These end-of-test namelists could differ if (a) there are multiple runs done within the same case directory (as is the case for ERS and ERI) or (b) for a test based on SystemTestCompareTwo, if case1 makes any changes to the namelist in `_case_one_setup` (because apparently namelist generation happens before that point).

I'd guess that the same issue applies if you use the `--compare` flag to `create_test` to compare against baselines generated with `bless_test_results`.

Solutions I can see, in order from least to most drastic, but also least to most helpful:

(1) Do not change any behavior, but add some documentation in `compare_test_results` and `bless_test_results` warning about these possible spurious NLCOMP failures.

(2) Along with the above warning, turn off namelist comparison by default in `compare_test_results`, because it can be more confusing than helpful. Maybe do the same for `bless_test_results`.

(3) Change the timing of namelist generation and comparison in create_test so that these are done at the end of the test. That way the operation would be the same whether you're doing in-test generation/comparison or generation/comparison via `bless_test_results`/`compare_test_results`. However, some people may like the fact that namelists are compared early in the test process, so they can notice namelist differences soon? (I don't personally rely on that.)

@jgfouca @fischer-ncar @cacraigucar (Cheryl, I think you sometimes use this) any thoughts on which of these we should do?


Discussion:
I would prefer 1 or 2 over 3.
@billsacks , this issue has been a thorn in my side for a long while. This is the reason that blessing namelists cannot be a simple file copy from the casedir to the baseline area; instead, we have to re-run preview_namelists and then copy the fresh nl files.

Another option would be to store the "fresh" version of namelists somewhere special in the case so they don't get "polluted" by the RUN phase. compare/bless would then use the fresh versions for their work. Unfortunately, that approach adds a fair bit of complexity to the system and could confuse users in different ways.

Of your three options, I would prefer (1) as well.
I prefer #1 as well as the current scripts work fine with CAM's workflow.  CAM uses create_test and does NOT use the --generate flag.  After all our testing is done and we are happy with the results, at that time we use bless_test_results to store the baseline information.  As @jgfouca stated, bless_test_results regenerates the baselines and stores the correct baseline for our comparison.  With our workflow, the namelist comparison is a useful tool and indicates the correct PASS/FAIL for our runs (even the ERS ones)

I would suggest while you are adding a warning to the documentation for bless_test_results, that it should also say that the baselines are regenerated before being compared.  This bit me when I was archiving a baseline for another user using bless_test_results and used a different code base than what the user had used (thinking that all I needed was a recent version of bless_test_results).  
Thanks, all. I'm fine with just (1) for now, then.

@cacraigucar can you please clarify this:

> I would suggest while you are adding a warning to the documentation for bless_test_results, that it should also say that the baselines are regenerated before being compared. This bit me when I was archiving a baseline for another user using bless_test_results and used a different code base than what the user had used (thinking that all I needed was a recent version of bless_test_results).

Is this just referring to namelists? So you're saying that bless_test_results runs preview_namelists before doing the baseline generation?
I should have been more specific.  bless_test_results appears to regenerate the namelists.   I defer the specifics of how it is doing this (and to confirm that it is indeed regenerating the namelists) to @jgfouca 

Resolving patch:
diff --git a/scripts/lib/CIME/case_cmpgen_namelists.py b/scripts/lib/CIME/case_cmpgen_namelists.py
--- a/scripts/lib/CIME/case_cmpgen_namelists.py
+++ b/scripts/lib/CIME/case_cmpgen_namelists.py
@@ -119,6 +119,13 @@ def case_cmpgen_namelists(case, compare=False, generate=False, compare_name=None
             output = ""
             if compare:
                 success, output = _do_full_nl_comp(case, test_name, compare_name, baseline_root)
+                if not success and ts.get_status(RUN_PHASE) is not None:
+                    run_warn = \
+"""NOTE: It is not necessarily safe to compare namelists after RUN
+phase has completed. Running a case can pollute namelists. The namelists
+kept in the baselines are pre-RUN namelists."""
+                    output += run_warn
+                    logging.info(run_warn)
             if generate:
                 _do_full_nl_gen(case, test_name, generate_name, baseline_root)
         except:


Test patch: