Repository: ioos/compliance-checker

Issue:
`Unidentifiable feature for variable {}` stored as message without throwing error/warning
# Issue

If a feature type cannot be identified, the message `Unidentifiable feature for variable XXX` is not generally printed by the compliance checker but only under certain circumstances.

# Description of the Issue

When the feature type of a data variable cannot be identified as message is appended to a `TestCtx` object (in l. 3233 of function `check_all_features_are_same_type` in file `cf.py`):

```
if feature is not None:
    feature_types_found[feature].append(name)
else:
    all_the_same.messages.append("Unidentifiable feature for variable {}"
                                 "".format(name))
```

`messages.append` is called to `all_the_same` but not `assert_true` as is done for other issues such as l. 3237 of function `check_all_features_are_same_type` in file `cf.py`:

```
all_the_same.assert_true(len(feature_types_found) < 2,
                         "Different feature types discovered in this dataset: {}"
                         "".format(feature_description))
```

As a result of this, the message only appears if another error or warning is thrown.


# Reproduction of the issue

Take either the two txt files or the two NetCDF files below. The txt files contain cdl definitions of two NetCDF datasets. The NetCDF files are generated from the cdl definitions. The file `file_test_iooscc_D1` will succeed. The file `file_test_iooscc_F1` will throw one error with two error messages (one of them `Unidentifiable feature for variable`). However, the variable, which causes `Unidentifiable feature for variable` is present in both files.

Files (`*.cdl` and `*.nc` could not be uploaded here; therefore, `*.txt` instead of `*.cdl` and `*.nc.gz` instead of `*.nc`):

* [file_test_iooscc_D1.txt](https://github.com/ioos/compliance-checker/files/3740151/file_test_iooscc_D1.txt)
* [file_test_iooscc_D1.nc.gz](https://github.com/ioos/compliance-checker/files/3740155/file_test_iooscc_D1.nc.gz)
* [file_test_iooscc_F1.txt](https://github.com/ioos/compliance-checker/files/3740157/file_test_iooscc_F1.txt)
* [file_test_iooscc_F1.nc.gz](https://github.com/ioos/compliance-checker/files/3740158/file_test_iooscc_F1.nc.gz)

To analyze the issue in more detail please use this Jupyter Notebook (change file ending to `.ipynb`)

[ioos_testing_github_issue.txt](https://github.com/ioos/compliance-checker/files/3740169/ioos_testing_github_issue.txt)

or this code snippet:

```
## Initialization
# load need IOOS Compliance Checker classes
from compliance_checker.runner import ComplianceChecker, CheckSuite

# initialize the checker
cs = CheckSuite()
cs.load_all_available_checkers()
cls = ComplianceChecker()

## First File -- No Message
# CASE D
checker_names = ['cf']
skip_checks=None
loc = 'file_test_iooscc_D1.cdl'
ds = cs.load_dataset(loc)
score_groups = cs.run(ds, skip_checks, *checker_names)

# Example Check: no message, no failed check
score_groups['cf'][0][22]

# Example Check: one message, no failed check
# >>>>>>> BUT: should fail (with LOW, MEDIUM OR HIGH priority) <<<<<<<
score_groups['cf'][0][23]

## Second File -- Two Messages
# CASE F
checker_names = ['cf']
skip_checks=None
loc = 'file_test_iooscc_F1.cdl'
ds = cs.load_dataset(loc)
score_groups = cs.run(ds, skip_checks, *checker_names)

# Example Check: no message, no failed check
score_groups['cf'][0][22]

# Example Check: one message, one failed check
# >>>>>>> BUT: should be two fails <<<<<<<
score_groups['cf'][0][23]

# see the two messages
score_groups['cf'][0][23].msgs
```


Discussion:
I am not sure, whether this is behavior is intended or not. If not, I can create a pull request fixing this.

In the latter case: which priority should the failing check get (high, medium or low)? Should the failing check be counted each time it fails? Or, should it be counted the first time and subsequent fails should only append their messages (but not increment the check counters)?
@neumannd, sorry for taking a little while looking at this.  In the event that there are unidentifiable features, the checker should keep the high priority per CF section 9.1 .  The unidentifiable features don't need to be deducting each time.  Once is enough.  the compliance these days still keeps the "score" under the hood, but we're mostly checking whether a given check "passes" (score == possible number of points)
Go ahead and make a candidate PR and I'll review it.
@benjwadams Thanks for the reply. I am not sure whether I got the last part correctly: Because the results are summarized on per-check-basis it does not matter whether I call `assert_true()` (at the particular instance of `TestCtx`) for each variable or don't call it anymore after the first `unidentifiable feature` -- right? In the case of the `standard_name`-checking the function `assert_true()` is called for each checked standard name. Hence, I would do it similarly in this case. OK?
Ideally, a message would appear for each unidentifiable feature.  Score is somewhat irrelevant, as long as `score < out_of` for error cases.

Resolving patch:
diff --git a/compliance_checker/cf/cf.py b/compliance_checker/cf/cf.py
--- a/compliance_checker/cf/cf.py
+++ b/compliance_checker/cf/cf.py
@@ -3225,13 +3225,18 @@ def check_all_features_are_same_type(self, ds):
         feature_types_found = defaultdict(list)
         for name in self._find_geophysical_vars(ds):
             feature = cfutil.guess_feature_type(ds, name)
-            # If we can't figure out the feature type, don't penalize, just
-            # make a note of it in the messages
+            # If we can't figure out the feature type, penalize. Originally, 
+            # it was not penalized. However, this led to the issue that the
+            # message did not appear in the output of compliance checker if
+            # no other error/warning/information was printed out for section
+            # 9.1.
+            found = False
             if feature is not None:
                 feature_types_found[feature].append(name)
-            else:
-                all_the_same.messages.append("Unidentifiable feature for variable {}"
-                                             "".format(name))
+                found = True
+            all_the_same.assert_true(found,
+                                     "Unidentifiable feature for variable {}"
+                                     "".format(name))
         feature_description = ', '.join(['{} ({})'.format(ftr, ', '.join(vrs)) for ftr, vrs in feature_types_found.items()])
 
         all_the_same.assert_true(len(feature_types_found) < 2,


Test patch: