Repository: ESMCI/cime

Issue:
Unexpected test results in multiinst tests if instances are not identical
I recently set up a new POP test to make sure multi-instance runs can handle different MARBL configurations, but I think the test results are misleading. I'm running an `SMS_N2` test, and the current behavior is

1. I run the test and generate baselines

   1. Namelists from all instances get copied to the baseline directory
   1. History files from instance `0001` get copied to the baseline directory

1. I run the test again and compare to existing baselines

   1. Namelists from each instance are compared to the existing baseline from the same instance, so all `NLCOMP` tests pass
   1. History files from each instance are compared to the baseline from instance `0001`, so instances that differ from the first (i.e. instance `0002`) fail the baseline comparison

So by definition, multi-instance tests are assuming that all instances are the same. I don't like that, but if that's the way it's going to be then I think a better test set-up would be to compare instance `0002` against instance `0001` so that the failure isn't reported as a baseline comparison failure -- the test run generating the baseline should also report this failure!

(A better solution, IMO, would be to copy history files from ALL instances to baselines and remove the assumption that they are all the same.)

From conversations in slack, it turns out this was partially discussed by @billsacks and @jedwards4b in #465 but that predates my usage of multi-instance tests so I didn't chime in at the time.

edit: as @billsacks points out, I somehow managed to type 406 instead of 465 in the last paragraph


Discussion:
For the record, I still support my feelings from https://github.com/ESMCI/cime/pull/465#discussion-diff-76501565 which are that we should generate and compare separately against all instances of the baselines, in order to support use cases like the one here.

@mnlevy1981 I think you meant to cite #465 rather than #406 ?
@jedwards4b , what's the reason not to utilize all the instances in baseline operations?
In tests up until now all instances are expected to get the same result.   I think that tests that expect different results in each instance are fundamentally different tests and should be given different names - that is SMS_N2 should always expect both to be the same.    
I disagree with @jedwards4b . There's nothing about SMS_N2 that says the two instances need to be the same, and I feel that it's very limiting to require that. NCK is a specific test type that says the two need to be the same, but we shouldn't impose that on other tests.

Philosophical differences on whether the two instances in SMS_N2 should be the same are somewhat irrelevant, though: The code in question applies to _all_ test types. So I believe that, even if we said, "Differences in the instances need to be tested via another test type", we'd still have to generalize the code in question so that that new test type would work correctly for baseline comparisons / generation.
I'll also add that this

> In tests up until now all instances are expected to get the same result.

is not a good reason to maintain current behavior. Our current tests of multi-instance are pretty weak right now because of this. It's just that nobody has taken the time to write the kinds of tests that we really should have for multi-instance - i.e., tests where the two instances differ. I am as much to blame as anyone else for this.
To parrot @billsacks, I also disagree with @jedwards4b statement

>  I think that tests that expect different results in each instance are fundamentally different tests and should be given different names - that is SMS_N2 should always expect both to be the same.

To me, `SMS_N2` just says "this is a smoke test with two instances", and there's no information about the configuration of the two instances in that. Also:

> In tests up until now all instances are expected to get the same result.

I understand why this has been the case up until now, but in my opinion it severely cripples the test suite when it comes to multi-instance because you are saying right up front "our multi-instance tests check to make sure that multiple instances run, but we aren't going to actually make sure the multi-instance configuration works as expected"... I can't think of any science applications where you really do want all the instances to be the same so it would be nice to make sure the infrastructure is actually setting up different instances correctly.
@billsacks, @mlevy @jedwards

I agree that it would be valuable to have the multi-instance tests have
each instance have different behavior. However, other than a baseline
comparison - how would you know that the test successfully passed? The main
purpose of our current multi-instance tests is to only demonstrate that
with multiple instances or a single instance the infrastructure preserves
answers. So its a simple sanity check. If suddenly you wanted to have
different configurations for the multi-instances - than what would the
smoke test achieve - other than as I mentioned - a valid comparison to
baselines - which is what Mike is trying to validate. From that point of
view I agree with Jim that we need different test names to distinguish
between tests that only have meaning via baseline comparisons or those
tests that provide sanity checks on the multi-instance configuration.



On Tue, Dec 19, 2017 at 12:25 PM, Michael Levy <notifications@github.com>
wrote:

> To parrot @billsacks <https://github.com/billsacks>, I also disagree with
> @jedwards4b <https://github.com/jedwards4b> statement
>
> I think that tests that expect different results in each instance are
> fundamentally different tests and should be given different names - that is
> SMS_N2 should always expect both to be the same.
>
> To me, SMS_N2 just says "this is a smoke test with two instances", and
> there's no information about the configuration of the two instances in
> that. Also:
>
> In tests up until now all instances are expected to get the same result.
>
> I understand why this has been the case up until now, but in my opinion it
> severely cripples the test suite when it comes to multi-instance because
> you are saying right up front "our multi-instance tests check to make sure
> that multiple instances run, but we aren't going to actually make sure the
> multi-instance configuration works as expected"... I can't think of any
> science applications where you really do want all the instances to be the
> same so it would be nice to make sure the infrastructure is actually
> setting up different instances correctly.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/ESMCI/cime/issues/2157#issuecomment-352860950>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AHlxE33AB8nxKxVJKqzY8uDmxv4ElM8Uks5tCA2VgaJpZM4RHO_2>
> .
>

Responding to @mvertens 

> I agree that it would be valuable to have the multi-instance tests have each instance have different behavior. However, other than a baseline comparison - how would you know that the test successfully passed?

How do you know that any test passes? Consider three POP `testmod` dirs:

1. `ecosys` -- POP / MARBL namelists have MARBL in it's default configuration
1. `no_source_sink` -- POP / MARBL namelists have MARBL in it's default configuration, except `lsource_sink = .false.`
1. `multiinst_ecosys` -- `user_nl_{pop,marbl}_0001` are identical to `user_nl_{pop,marbl}` from `ecosys` and `user_nl_{pop,marbl}_0002` are identical to `user_nl_{pop,marbl}` from `no_source_sink`

What's the difference between

* running both an `SMS.____.pop-ecosys` and an `SMS._____.pop-no_source_sink` test, and
* running a single `SMS_N2.____.pop-multiinst_ecosys` test (assuming the baselines for both instances are kept)

If the latter test results in a `BASELINE` failure you have a little more work to do in the logs to figure out what exactly didn't match, and if the run itself crashes you have a lot more work to do to figure out why, but other than that I'm not seeing a difference. If you aren't comparing to a baseline and the `SMS_N2` run does not crash, you have exactly as much information as if the two separate `SMS` tests run without crashing. I'm not advocating shoe-horning as many tests as possible into one gigantic multi-instance test, but I do think the `SMS_N2` approach is a good way to test multi-instance beyond just "am I creating two instances"?

> The main purpose of our current multi-instance tests is to only demonstrate that with multiple instances or a single instance the infrastructure preserves answers. So its a simple sanity check.

I think the fact that the instance count is an option available to all (most?) tests means that there isn't a "current multi-instance test"; just the ability to run a smoke test with 1 instance or multiple, the ability to run a restart test with 1 instance or multiple, etc. Much like I can opt not to put a `user_nl_marbl` file in a testmod directory and POP will run MARBL with its default configuration, I can opt not to put `user_nl_marbl_####` files in a testmod directory and all instances of a multi-instance tests will run MARBL with its "default" configuration.

> I agree with Jim that we need different test names to distinguish between tests that only have meaning via baseline comparisons or those tests that provide sanity checks on the multi-instance configuration

I disagree, I think you distinguish between multi-instance tests with meaningful baseline comparisons vs those that are meant to provide sanity checks on the infrastructure by looking at the testmod directory behind the tests.

In my mind, a good use for "we need a new test beyond a simple smoke test" would be something like the `NCK` test (enabling multi-instance shouldn't change answers, so let's run a single instance test and then configure both instances of a two-instance test in exactly the same way as that single run and make sure answers haven't changed) or a test where we want to compare history files between the two instances (maybe the second instance adds additional diagnostics without changing answers, e.g. instance one is a default configuration and instance two adds some passive tracers that should not feedback to existing prognostic variables<sup>*</sup>).

<sup>*</sup> For the record, POP's current implementation of multi-instance does not support this theoretical test (both instances of POP need to be built with the same tracer count / tracer modules) so it's included solely as an example and not a wish-list item.
@mnlevy1981 - @mvertens and I talked and we're on the same page in terms of the need to copy and compare all instances when doing baseline comparison / generation. However, @jedwards4b has a lot on his plate right now. Is this something you'd be able to take on? I think if you look back at the discussion on #465 you should see at least some of what needs to be changed... I suspect that all changes would be restricted to `hist_utils.py`.

Replying to some of the other points that have been raised:

First: Much of this discussion is somewhat tangential to this issue: Regardless of how we decide to test multi-instance, it's still going to be important to fix the issue that baseline generation is done only on the first instance.

I agree with sentiment that we should have a test type that covers multi-instance more rigorously.  I believe this could be accomplished via a new test type that: (1) Runs two single-instance cases, where the instances differ in some way; (2) Runs a multi-instance case where instance 1 matches the first case in (1) and instance 2 matches the second case in (2); then compares both instances from (1) and (2). This would probably need to be done in conjunction with component-specific testmods (as @mnlevy1981 is doing for his _N2 tests).

As for the utility of `_N2`: I'd argue that this is still quite useful, and more so if the instances differ in some way. For example, `ERS_N2` is useful to ensure that multi-instance restarts correctly, and can be used to ensure that the two instances write and then read the correct restart file. (As with other multi-instance tests, this test is weaker if the two instances are identical, because it could be that both instances are reading from the `0001` restart file, for example.)

Resolving patch:
diff --git a/scripts/lib/CIME/hist_utils.py b/scripts/lib/CIME/hist_utils.py
--- a/scripts/lib/CIME/hist_utils.py
+++ b/scripts/lib/CIME/hist_utils.py
@@ -403,16 +403,6 @@ def generate_baseline(case, baseline_dir=None, allow_baseline_overwrite=False):
             if os.path.exists(baseline):
                 os.remove(baseline)
 
-            #special care for multi-instance cases,
-            #only keep first instance and
-            #remove instance string from filename
-            m = re.search("(.*%s.*)_([0-9]{4})(.h.*)" % model, baseline)
-            if m is not None:
-                if m.group(2) != '0001':
-                    continue
-                baseline = m.group(1)+m.group(3)
-
-                logger.debug("Found multiinstance hist file {}".format(hist))
             shutil.copy(hist, baseline)
             comments += "    generating baseline '{}' from file {}\n".format(baseline, hist)
 


Test patch: