Repository: ioos/compliance-checker

Issue:
skip-checks option doesn't work as documented
**Version of compliance checker running:** 5.x

**Describe the checker this affects:**
Core checker functionality, i.e., affects any check suite.

**Attach a minimal CDL or NetCDF file which is able to reproduce the issue**
Any file that fails a specific check method. e.g. [this file](https://s3-ap-southeast-2.amazonaws.com/imos-data/IMOS/eMII/test_files/ANMN/AM/IMOS_ANMN-AM_GST_20190419T100000Z_NRSMAI_FV01_NRSMAI-CO2-1904-delayed_END_20191002T040000Z_C-20210525T140027Z.nc) is missing the recommended `history` attribute, so is fails the `check_convention_globals` check in the CF check suite.

**To Reproduce:**
For the example file above, if I want to ignore that specific CF recommendation and have the checker pass the file, I could skip the `check_convention_globals`. The help text from the command-line interface describes this option:
```
  --skip-checks SKIP_CHECKS, -s SKIP_CHECKS
                        Specifies tests to skip. Can take the form of either `<check_name>` or
                        `<check_name>:<skip_level>`. The first form skips any checks matching the
                        name. In the second form <skip_level> may be specified as "A", "M", or
                        "L". "A" skips all checks and is equivalent to calling the first form. "M"
                        will only show high priority output from the given check and will skip
                        medium and low. ...
```
However, the simple `<check_name>` form doesn't work anymore (it did with version 4.x):
```
$ compliance-checker -t cf:1.6 -s check_convention_globals IMOS_ANMN-AM_GST_20190419T100000Z_NRSMAI_FV01_NRSMAI-CO2-1904-delayed_END_20191002T040000Z_C-20210525T140027Z.nc 

...
                                    Warnings                                    
--------------------------------------------------------------------------------
§2.6 Attributes
* §2.6.2 global attribute history should exist and be a non-empty string
```


**Describe the issue below:**
Specifying the second form of the option (`-s check_convention_globals:M`), does skip the check as expected.
For context, we rely on this functionality in cases where it is not possible/practical to make a file fully compliant, or there is a bug in the checker that falsely rejects certain files, but we want to apply the rest of the checks to all files passing through our data ingenstion pipelines.
skip-checks option doesn't work as documented
**Version of compliance checker running:** 5.x

**Describe the checker this affects:**
Core checker functionality, i.e., affects any check suite.

**Attach a minimal CDL or NetCDF file which is able to reproduce the issue**
Any file that fails a specific check method. e.g. [this file](https://s3-ap-southeast-2.amazonaws.com/imos-data/IMOS/eMII/test_files/ANMN/AM/IMOS_ANMN-AM_GST_20190419T100000Z_NRSMAI_FV01_NRSMAI-CO2-1904-delayed_END_20191002T040000Z_C-20210525T140027Z.nc) is missing the recommended `history` attribute, so is fails the `check_convention_globals` check in the CF check suite.

**To Reproduce:**
For the example file above, if I want to ignore that specific CF recommendation and have the checker pass the file, I could skip the `check_convention_globals`. The help text from the command-line interface describes this option:
```
  --skip-checks SKIP_CHECKS, -s SKIP_CHECKS
                        Specifies tests to skip. Can take the form of either `<check_name>` or
                        `<check_name>:<skip_level>`. The first form skips any checks matching the
                        name. In the second form <skip_level> may be specified as "A", "M", or
                        "L". "A" skips all checks and is equivalent to calling the first form. "M"
                        will only show high priority output from the given check and will skip
                        medium and low. ...
```
However, the simple `<check_name>` form doesn't work anymore (it did with version 4.x):
```
$ compliance-checker -t cf:1.6 -s check_convention_globals IMOS_ANMN-AM_GST_20190419T100000Z_NRSMAI_FV01_NRSMAI-CO2-1904-delayed_END_20191002T040000Z_C-20210525T140027Z.nc 

...
                                    Warnings                                    
--------------------------------------------------------------------------------
§2.6 Attributes
* §2.6.2 global attribute history should exist and be a non-empty string
```


**Describe the issue below:**
Specifying the second form of the option (`-s check_convention_globals:M`), does skip the check as expected.
For context, we rely on this functionality in cases where it is not possible/practical to make a file fully compliant, or there is a bug in the checker that falsely rejects certain files, but we want to apply the rest of the checks to all files passing through our data ingenstion pipelines.


Discussion:

Resolving patch:
diff --git a/compliance_checker/suite.py b/compliance_checker/suite.py
--- a/compliance_checker/suite.py
+++ b/compliance_checker/suite.py
@@ -345,23 +345,8 @@ def _process_skip_checks(cls, skip_checks):
                         )
                     )
                     check_max_level = BaseCheck.HIGH
-                else:
-                    try:
-                        check_max_level = check_lookup[split_check_spec[1]]
-                    except KeyError:
-                        warnings.warn(
-                            "Skip specifier '{}' on check '{}' not found,"
-                            " defaulting to skip entire check".format(
-                                split_check_spec[1], check_name
-                            )
-                        )
-                        check_max_level = BaseCheck.HIGH
 
-                check_dict[check_name] = check_max_level
-        else:
-            for check_name in skip_checks:
-                # always process
-                check_dict[check_name] = 0
+            check_dict[check_name] = check_max_level
 
         return check_dict
 


Test patch:
diff --git a/compliance_checker/tests/test_suite.py b/compliance_checker/tests/test_suite.py
--- a/compliance_checker/tests/test_suite.py
+++ b/compliance_checker/tests/test_suite.py
@@ -8,6 +8,7 @@
 
 from compliance_checker.base import BaseCheck, GenericFile, Result
 from compliance_checker.suite import CheckSuite
+from compliance_checker.acdd import ACDDBaseCheck
 
 static_files = {
     "2dim": resource_filename("compliance_checker", "tests/data/2dim-grid.nc"),
@@ -106,10 +107,14 @@ def test_skip_checks(self):
         ds = self.cs.load_dataset(static_files["2dim"])
         # exclude title from the check attributes
         score_groups = self.cs.run_all(ds, ["acdd"], skip_checks=["check_high"])
-        assert all(
-            sg.name not in {"Conventions", "title", "keywords", "summary"}
-            for sg in score_groups["acdd"][0]
-        )
+        msg_set = {msg
+                   for sg in score_groups["acdd"][0]
+                   for msg in sg.msgs
+                   if sg.weight == BaseCheck.HIGH
+                   }
+        skipped_messages = {att + " not present" for att in ACDDBaseCheck().high_rec_atts}
+        # none of the skipped messages should be in the result set
+        self.assertTrue(len(msg_set & skipped_messages) == 0)
 
     def test_skip_check_level(self):
         """Checks level limited skip checks"""