Repository: ioos/compliance-checker

Issue:
Update check_cell_measures() function in cf_1_6.y
added a test to verify units are recognized by UDUNITS in the check_cell_measures() function
Single valued `flag_values` causes an Exception
**Version of compliance checker running:**
5.0.2.dev90+gcad1a7c
https://github.com/ioos/compliance-checker/tree/cad1a7c9900459fd33a403e58d79868a54d907b7

**Describe the checker this affects:**
CF

**Attach a minimal CDL or NetCDF file which is able to reproduce the issue**
```
netcdf singleflag {
variables:
  int x ;
    x:flag_values = -1 ;
    x:flag_meanings = "any" ;

}
```

**To Reproduce:**
```
$ ncgen -k3 singleflag.cdl.txt -o singleflag.nc
$ compliance-checker -vvv -t=cf singleflag.nc
Running Compliance Checker on the datasets from: ['singleflag.nc']
   . . .
* §2.6.2 global attribute history should exist and be a non-empty string
* §2.6.1 Conventions field is not present
WARNING: The following exceptions occurred during the cf checker (possibly indicate compliance checker issues):
cf.check_flags: 'numpy.int32' object is not iterable
  File "/net/home/h05/itpp/git/ioos_compliance-checker/compliance_checker/cf/cf_1_6.py", line 1072, in check_flags
    valid_values = self._check_flag_values(ds, name)
  File "/net/home/h05/itpp/git/ioos_compliance-checker/compliance_checker/cf/cf_1_6.py", line 1125, in _check_flag_values
    flag_set = set(flag_values)

$ 
```

**Describe the issue below:**
The error occurs trying to iterate over 'flag_values', but the code requires also the presence of a 'flag_meanings' to hit the error.

The CF validity of the example is, I think, questionable :  Is a single flag value/meaning acceptable or not?
(update : on this, see https://github.com/ioos/compliance-checker/issues/914#issuecomment-1089513214 and https://github.com/ioos/compliance-checker/issues/914#issuecomment-1089563178 )
Having said that, I did have a real file from another person which contained this.
Arguably, the checker code needs to be more robust.


Discussion:

Possibly related :
  * https://github.com/ioos/compliance-checker/issues/792
  * https://github.com/ioos/compliance-checker/issues/914
    * fixed in https://github.com/ioos/compliance-checker/pull/915  : we probably want something simliar (though I think this one ins simpler ?)
I can probably propose a fix, but someone else may be better placed.
Would anyone else agree that this is worth fixing ?
Yes, this is worth fixing.  NetCDF's data model treats attributes as an array of a certain type.   NetCDF4-Python will take a single element attribute and cast it to scalar.

Resolving patch:
diff --git a/compliance_checker/cf/cf_1_6.py b/compliance_checker/cf/cf_1_6.py
--- a/compliance_checker/cf/cf_1_6.py
+++ b/compliance_checker/cf/cf_1_6.py
@@ -1078,14 +1078,21 @@ def check_flags(self, ds):
                 ret_val.append(valid_masks)
 
             if flag_values is not None and flag_masks is not None:
-                allv = list(
-                    map(lambda a, b: a & b == a, list(zip(flag_values, flag_masks)))
-                )
+                vals_arr = np.array(flag_values, ndmin=1)
+                masks_arr = np.array(flag_masks, ndmin=1)
+                # IMPLEMENTATION CONFORMANCE 3.5 RECOMMENDED 1/1
+                # If shapes aren't equal, we can't do proper elementwise
+                # comparison
+                if vals_arr.size != masks_arr.size:
+                    allv = False
+                else:
+                    allv = np.all(vals_arr & masks_arr == vals_arr)
 
-                allvr = Result(BaseCheck.MEDIUM, all(allv), self.section_titles["3.5"])
+                allvr = Result(BaseCheck.MEDIUM, allv,
+                               self.section_titles["3.5"])
                 if not allvr.value:
                     allvr.msgs = [
-                        "flag masks and flag values for '{}' combined don't equal flag value".format(
+                        "flag masks and flag values for '{}' combined don't equal flag values".format(
                             name
                         )
                     ]
@@ -1121,9 +1128,9 @@ def _check_flag_values(self, ds, name):
 
 
         # the flag values must be independent, no repeating values
-        flag_set = set(flag_values)
+        flag_set = np.unique(flag_values)
         valid_values.assert_true(
-            len(flag_set) == np.array(flag_values).size,
+            flag_set.size == np.array(flag_values).size,
             "{}'s flag_values must be independent and can not be repeated".format(name),
         )
 
@@ -1167,7 +1174,7 @@ def _check_flag_masks(self, ds, name):
 
         valid_masks.assert_true(
             variable.dtype.type == flag_masks.dtype.type,
-            "flag_masks ({}) mustbe the same data type as {} ({})"
+            "flag_masks ({}) must be the same data type as {} ({})"
             "".format(flag_masks.dtype.type, name, variable.dtype.type),
         )
 
@@ -1177,6 +1184,11 @@ def _check_flag_masks(self, ds, name):
             or np.issubdtype(variable.dtype, "b")
         )
 
+        valid_masks.assert_true(0 not in np.array(flag_masks),
+                                f"flag_masks for variable {variable.name} must "
+                                 "not contain zero as an element")
+
+
         valid_masks.assert_true(
             type_ok,
             "{}'s data type must be capable of bit-field expression".format(name),


Test patch:
diff --git a/compliance_checker/tests/test_cf.py b/compliance_checker/tests/test_cf.py
--- a/compliance_checker/tests/test_cf.py
+++ b/compliance_checker/tests/test_cf.py
@@ -483,7 +483,8 @@ def test_check_standard_name(self):
         temperature.standard_name = "sea_water_temperature"
         temperature.ancillary_variables = "temperature_flag"
 
-        temperature_flag = dataset.createVariable("temperature_flag", "i2", ("time",))
+        temperature_flag = dataset.createVariable("temperature_flag", "i2",
+                                                  ("time",))
         # bad modifier
         temperature_flag.standard_name = "sea_water_temperature status flag"
         _, _, messages = get_results(self.cf.check_standard_name(dataset))
@@ -677,6 +678,10 @@ def test_check_flags(self):
         # only 4 variables in this dataset do not have perfect scores
         imperfect = [r.value for r in results if r.value[0] < r.value[1]]
         assert len(imperfect) == 4
+        dataset.variables["conductivity_qc"] = MockVariable(dataset.variables["conductivity_qc"])
+        # Test with single element.  Will fail, but should not throw exception.
+        dataset.variables["conductivity_qc"].flag_values = np.array([1], dtype=np.int8)
+        results = self.cf.check_flags(dataset)
 
     def test_check_flag_masks(self):
         dataset = self.load_dataset(STATIC_FILES["ghrsst"])
@@ -690,9 +695,24 @@ def test_check_flag_masks(self):
         flags_var = dataset.createVariable("flags", "f8", ("time",))
         flags_var.standard_name = "quality_flag"
         flags_var.flag_meanings = "LAND"
+        flags_var.flag_values = [1]
+        # test single element
         flags_var.flag_masks = np.array([1], dtype="i2")
         results = self.cf.check_flags(dataset)
         assert scored > 0 and scored == out_of
+        # TEST CONFORMANCE 3.5 REQUIRED 7/8
+        flags_var.flag_masks = np.array([0, 1], dtype="i2")
+        results = self.cf.check_flags(dataset)
+        score, out_of, messages = get_results(results)
+        assert ("flag_masks for variable flags must not contain zero as an "
+                "element" in messages)
+        # IMPLEMENTATION 3.5 REQUIRED 1/1
+        flags_var.flag_masks = np.array([1], dtype="i2")
+        flags_var.flag_values = np.array([2], dtype="i2")
+        results = self.cf.check_flags(dataset)
+        score, out_of, messages = get_results(results)
+        assert ("flag masks and flag values for 'flags' combined don't equal "
+                "flag values" in messages)
 
     def test_check_bad_units(self):
         """Load a dataset with units that are expected to fail (bad_units.nc).