Repository: ESMCI/cime

Issue:
Tests fail to report errors if comparison between the two cases fails
In my testing with cime5.0.7, I discovered that tests fail to report errors even if the comparison between the two cases shows differences.

Specifically, I ran the ERS test in https://github.com/ESMCI/cime/issues/295 out of the cesm2 alpha01 sandbox, and found that it reports:

```
PASS ERS_Ld7.f10_f10.ICLM45.yellowstone_intel COMPARE_base_rest
```

despite:

```
[yslogin5:/glade/scratch/sacks/ERS_Ld7.f10_f10.ICLM45.yellowstone_intel.20160811_032808/run]$ tail ERS_Ld7.f10_f10.ICLM45.yellowstone_intel.20160811_032808.cpl.hi.0001-01-08-00000.nc.rest.cprnc.out

SUMMARY of cprnc:
 A total number of    182 fields were compared
          of which     39 had non-zero differences
               and      0 had differences in fill patterns
               and      0 had different dimension sizes
 A total number of      0 fields could not be analyzed
 A total number of      0 fields on file 1 were not found on file2.
  diff_test: the two files seem to be DIFFERENT
```

It looks like the problem is that, in the recent test refactor for rerunnability, _component_compare_test was changed to:

```
        status = TEST_PASS_STATUS if rc == 0 else TEST_FAIL_STATUS
```

But it looks like differences between the files does NOT generate a non-zero error code: A non-zero error code is only generated for errors in the script, I think.

At a glance, it looks like _compare_baseline has the same bug.



Discussion:
@mvertens @jedwards4b @fischer-ncar : This is a critical bug that needs to be resolved before beta01 is made.

@sacsk:  Thanks for catching this. I am really concerned at this point
about the robustness of the testing we are doing for this beta tag. I agree
that this is a critical bug and needs to be resolved. But as a first step
we need to manually check the output of all the cprnc.out files to verify
what functional and baseline failures we are detecting. Is this a
functionality failure you would have expected for this ERS test?

On Wed, Aug 10, 2016 at 9:59 PM, Bill Sacks notifications@github.com
wrote:

> @mvertens https://github.com/mvertens @jedwards4b
> https://github.com/jedwards4b @fischer-ncar
> https://github.com/fischer-ncar : This is a critical bug that needs to
> be resolved before beta01 is made.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> https://github.com/ESMCI/cime/issues/380#issuecomment-239066946, or mute
> the thread
> https://github.com/notifications/unsubscribe-auth/AHlxE77laghNb993BWZx2Z2Vfw-JKghkks5qep4kgaJpZM4Jhx3R
> .

@mvertens : Sorry I wasn't more clear in my bug report: I think you're asking whether I expected this ERS test to pass or fail. The answer is that I expected it to fail: I deliberately introduced an error in the CLM code to try to force a failure.

I will add: Thanks to @jgfouca 's recent rework, it's now less likely that there will be problems in individual test cases: As long as (1) a test case actually calls _component_compare_test, and (2) we verify that _component_compare_test is working correctly in general, then we can be pretty confident that that test will report results correctly (in contrast to earlier versions, where each test case needed to have the proper logic to test the return code of _component_compare_test). That is to say: Our efforts should be on verifying correctness of _component_compare_test and _compare_baseline in some general cases.

It looks like baseline comparisons are correct, at least from looking at the code (I haven't tested this myself): component_compgen_baseline.sh has this code at the bottom:

``` bash
    if [[ $overall_compare_status == PASS ]]; then
        exit 0
    else
        exit 2
    fi
```

Maybe the solution, then, is to put similar code at the bottom of component_compare_test.sh?


Resolving patch:
diff --git a/utils/python/CIME/XML/env_mach_specific.py b/utils/python/CIME/XML/env_mach_specific.py
--- a/utils/python/CIME/XML/env_mach_specific.py
+++ b/utils/python/CIME/XML/env_mach_specific.py
@@ -289,9 +289,9 @@ def get_module_system_type(self):
 
     def get_module_system_init_path(self, lang):
         init_nodes = self.get_optional_node("init_path", attributes={"lang":lang})
-        return init_nodes.text if init_nodes else None
+        return init_nodes.text if init_nodes is not None else None
 
     def get_module_system_cmd_path(self, lang):
         cmd_nodes = self.get_optional_node("cmd_path", attributes={"lang":lang})
-        return cmd_nodes.text if cmd_nodes else None
+        return cmd_nodes.text if cmd_nodes is not None else None
 


Test patch:
diff --git a/scripts/Tools/component_compare_test.sh b/scripts/Tools/component_compare_test.sh
--- a/scripts/Tools/component_compare_test.sh
+++ b/scripts/Tools/component_compare_test.sh
@@ -394,4 +394,18 @@ else
     print_status "$overall_status" "$compare_info: test functionality summary ($msg)" "${testcase_base}"
 fi
 
+if [[ $overall_status == FAIL ]]; then
+  exit 1
+else
+  exit 0
+fi
+
+
+
+
+
+
+
+
+