anon-iclr-submission/benchname-bug-localization huggingface dataset:
Repository: googlecloudplatform/dataproc-templates
Issue: [python][bug] JDBC to GCS csv headers not populating

CSV output using jdbc to gcs template is not populating headers.
Fix Line 195 in code change .option(constants.FORMAT_CSV, True) to .option(constants.HEADER, True)

Changed files:
python/dataproc_templates/jdbc/jdbc_to_gcs.py

diff --git a/python/dataproc_templates/jdbc/jdbc_to_gcs.py b/python/dataproc_templates/jdbc/jdbc_to_gcs.py
index ccbec7e0..35ad017a 100644
--- a/python/dataproc_templates/jdbc/jdbc_to_gcs.py
+++ b/python/dataproc_templates/jdbc/jdbc_to_gcs.py
@@ -192,7 +192,7 @@ class JDBCToGCSTemplate(BaseTemplate):
                 .save(output_location)
         elif output_format == constants.FORMAT_CSV:
             writer \\
-                .option(constants.FORMAT_CSV, True) \\
+                .option(constants.HEADER, True) \\
                 .csv(output_location)
         elif output_format == constants.FORMAT_JSON:
             writer \\
Repository: ioos/compliance-checker
Issue: Fail result for `Flags and flag attributes` check with no reasoning

Running the CF checker on [this file](http://thredds.aodn.org.au/thredds/fileServer/IMOS/SRS/SST/ghrsst/L3S-1d/dn/2016/20160919092000-ABOM-L3S_GHRSST-SSTfnd-AVHRR_D-1d_dn.nc) results in 

```
Name                             Priority:     Score:Reasoning
--------------------------------------------------------------------------------
§3.5 Flags and flag attributes         :3:    31/32 :  
```

An apparently failed result, with no reasoning message.


Changed files:
compliance_checker/cf/cf.py

diff --git a/compliance_checker/cf/cf.py b/compliance_checker/cf/cf.py
index cbc4b19..9bfffda 100644
--- a/compliance_checker/cf/cf.py
+++ b/compliance_checker/cf/cf.py
@@ -926,12 +926,15 @@ class CFBaseCheck(BaseCheck):
                 ret_val.append(Result(BaseCheck.HIGH, (ok_count, 2), '§3.5 Flags and flag attributes', msgs=msgs))
 
                 # 7) the flag_masks attribute values must be non-zero
-                zeros = [x for x in flag_masks if x == 0]
+                zeros_absent = 0 not in flag_masks
+
                 msgs = []
-                if len(zeros):
+                if not zeros_absent:
                     msgs = ['flag_masks attribute values contains a zero']
 
-                ret_val.append(Result(BaseCheck.HIGH, len(zeros) != 0, '§3.5 Flags and flag attributes', msgs=msgs))
+                ret_val.append(Result(BaseCheck.HIGH, zeros_absent,
+                                      '§3.5 Flags and flag attributes',
+                                      msgs=msgs))
 
             # 9) when both defined, boolean AND of each entry in flag_values with corresponding entry in flag_masks
             #    should equal the flags_value entry
Repository: ioos/compliance-checker
Issue: CF Checker records omission of “reference” global attribute as an error

Originally part of #501

(8) IOOS checker records omission of “reference” global attribute as an error: use of this attribute is neither required nor recommended.

Changed files:
compliance_checker/cf/cf.py, compliance_checker/tests/test_cf.py

diff --git a/compliance_checker/cf/cf.py b/compliance_checker/cf/cf.py
index 14e6e89..915fde8 100644
--- a/compliance_checker/cf/cf.py
+++ b/compliance_checker/cf/cf.py
@@ -800,6 +800,8 @@ class CFBaseCheck(BaseCheck):
         :rtype: list
         :return: List of Results
         """
+        # The attrs are optional and only needs to be a string and non-empty if it
+        # exists.
         attrs = ['institution', 'source', 'references', 'comment']
 
         valid_attributes = TestCtx(BaseCheck.MEDIUM, '§2.6.2 Recommended Attributes')
@@ -826,17 +828,6 @@ class CFBaseCheck(BaseCheck):
                                              "{} global attribute should be a non-empty string"
                                              "".format(attribute))
                 attr_bin.add(attribute)
-        # Make sure we've seen each attribute at least once.
-
-        valid_attributes.assert_true('institution' in attr_bin,
-                                     "institution should be defined")
-        valid_attributes.assert_true('source' in attr_bin,
-                                     "source should be defined")
-        valid_attributes.assert_true('references' in attr_bin,
-                                     "references should be defined")
-
-        # comment is optional and only needs to be a string and non-empty if it
-        # exists.
 
         return valid_attributes.to_result()
 
diff --git a/compliance_checker/tests/test_cf.py b/compliance_checker/tests/test_cf.py
index 5434bf3..36730c1 100644
--- a/compliance_checker/tests/test_cf.py
+++ b/compliance_checker/tests/test_cf.py
@@ -247,7 +247,7 @@ class TestCF(BaseTestCase):
         # 1x institution
         # 1x source
         # 1x EMPTY references
-        assert result.value == (15, 16)
+        assert result.value == (12, 13)
         assert result.msgs[0] == "references global attribute should be a non-empty string"
 
         dataset = self.load_dataset(STATIC_FILES['bad_data_type'])
@@ -256,10 +256,8 @@ class TestCF(BaseTestCase):
         # institution is a 10L
         # no source
         # comments doment matter unless they're empty
-        assert result.value == (1, 4)
+        assert result.value == (0, 1)
         assert result.msgs[0] == 'salinity:institution should be a non-empty string'
-        assert result.msgs[1] == 'source should be defined'
-        assert result.msgs[2] == 'references should be defined'
 
     def test_check_standard_name(self):
         """
Repository: ioos/compliance-checker
Issue: ACDD checks for "acknowledgement", IOOS checks for "acknowledgment"

When running "cchecker.py -t acdd:latest", it looks for either "acknowledgment" (US spelling, from deprecated UDDC convention) or "acknowledgement" (UK spelling, as required by superceding ACDD convention). However, when running "cchecker.py -t ioos:latest", the test only looks for the deprecated US spelling. It should either look for both spellings like the ACDD test or switch to the UK spelling as currently required.

Example:
http://www.pacioos.hawaii.edu/archive/WQB04agg/WQB04agg.2010.nc

Snippet for cchecker.py output:

Platform Sponsor                       :3:     0/ 1 : Attr 'acknowledgment'
                                                      (IOOS concept: 'Platform
                                                      Sponsor') not found in
                                                      dataset

Changed files:
compliance_checker/ioos.py

diff --git a/compliance_checker/ioos.py b/compliance_checker/ioos.py
index ba292ba..115d3a2 100644
--- a/compliance_checker/ioos.py
+++ b/compliance_checker/ioos.py
@@ -52,7 +52,7 @@ class IOOSNCCheck(BaseNCCheck, IOOSBaseCheck):
         Check all global NC attributes for existence.
         """
         return [
-            self._has_attr(ds, 'acknowledgment', 'Platform Sponsor'),
+            self._has_attr(ds, 'acknowledgement', 'Platform Sponsor'),
             self._has_attr(ds, 'publisher_email', 'Station Publisher Email'),
             self._has_attr(ds, 'publisher_email', 'Service Contact Email', BaseCheck.MEDIUM),
             self._has_attr(ds, 'institution', 'Service Provider Name', BaseCheck.MEDIUM),
Repository: ioos/compliance-checker
Issue: CDL check fails on text files (py3)

The recent CDL additions (#297) have broken the ability to pass a text file on the command line (IE a getcaps/describesensor xml document) on python 3 environments:

```
~/dev/compliance-checker -> python cchecker.py --test ioos ~/Downloads/aoos_sos_getcaps.xml
Running Compliance Checker on the dataset from: /home/daf/Downloads/aoos_sos_getcaps.xml
Traceback (most recent call last):
  File "cchecker.py", line 79, in <module>
    sys.exit(main())
  File "cchecker.py", line 68, in main
    args.format)
  File "/home/daf/dev/compliance-checker/compliance_checker/runner.py", line 44, in run_checker
    ds = cs.load_dataset(ds_loc)
  File "/home/daf/dev/compliance-checker/compliance_checker/suite.py", line 576, in load_dataset
    elif is_cdl_file(ds_str, first_chunk):
  File "/home/daf/dev/compliance-checker/compliance_checker/suite.py", line 567, in is_cdl_file
    if data.startswith('netcdf') or 'dimensions' in data:
TypeError: startswith first arg must be bytes or a tuple of bytes, not str
```


Changed files:
compliance_checker/suite.py

diff --git a/compliance_checker/suite.py b/compliance_checker/suite.py
index d5eaaad..c0ed5ad 100644
--- a/compliance_checker/suite.py
+++ b/compliance_checker/suite.py
@@ -482,9 +482,9 @@ class CheckSuite(object):
         Attempt to parse an xml string conforming to either an SOS or SensorML
         dataset and return the results
         """
-        xml_doc = ET.fromstring(str(doc))
+        xml_doc = ET.fromstring(doc)
         if xml_doc.tag == "{http://www.opengis.net/sos/1.0}Capabilities":
-            ds = SensorObservationService(ds_str, xml=str(doc))
+            ds = SensorObservationService(None, xml=doc)
 
         elif xml_doc.tag == "{http://www.opengis.net/sensorML/1.0.1}SensorML":
             ds = SensorML(xml_doc)
@@ -524,7 +524,7 @@ class CheckSuite(object):
                 r = requests.get(ds_str)
                 r.raise_for_status()
 
-                doc = r.text
+                doc = r.content
             else:
                 raise Exception("Could not understand response code %s and content-type %s" % (rhead.status_code, rhead.headers.get('content-type', 'none')))
         else:
@@ -564,7 +564,7 @@ class CheckSuite(object):
                 '''
                 if os.path.splitext(filename)[-1] == '.cdl':
                     return True
-                if data.startswith('netcdf') or 'dimensions' in data:
+                if data.startswith(b'netcdf') or b'dimensions' in data:
                     return True
                 return False
 
@@ -577,7 +577,7 @@ class CheckSuite(object):
                     ds_str = self.generate_dataset(ds_str)
                 else:
                     f.seek(0)
-                    doc = "".join(f.readlines())
+                    doc = f.read()
 
         if doc is not None:
             ds = self.process_doc(doc)
Repository: ioos/compliance-checker
Issue: JSON output missing argument

Attempting to run compliance-checker on Python 2 or 3 with JSON output fails due to a missing function argument.

```
compliance-checker --test ioos -f json -c strict compliance_checker/tests/data/conv_bad.nc                                                                    
Traceback (most recent call last):
  File "/home/badams_local/.virtualenvs/cchecker_py3/bin/compliance-checker", line 9, in <module>
    load_entry_point('compliance-checker', 'console_scripts', 'compliance-checker')()
  File "/home/badams_local/devel/compliance-checker/cchecker.py", line 41, in main
    args.format)
  File "/home/badams_local/devel/compliance-checker/compliance_checker/runner.py", line 50, in run_checker
    groups = cls.json_output(cs, score_groups, output_filename, ds_loc, limit)
  File "/home/badams_local/devel/compliance-checker/compliance_checker/runner.py", line 116, in json_output
    cs.json_output(checker, groups, f, ds_loc)
TypeError: json_output() missing 1 required positional argument: 'limit'
```


Changed files:
compliance_checker/tests/test_cli.py, compliance_checker/suite.py

diff --git a/compliance_checker/suite.py b/compliance_checker/suite.py
index e2f683e..ce01b07 100644
--- a/compliance_checker/suite.py
+++ b/compliance_checker/suite.py
@@ -225,7 +225,7 @@ class CheckSuite(object):
         aggregates = self.build_structure(check_name, groups, source_name, limit)
         aggregates = self.serialize(aggregates)
         json_string = json.dumps(aggregates, ensure_ascii=False)
-        file_object.write(str(json_string))
+        file_object.write(json_string)
         return
 
     def serialize(self, o):
diff --git a/compliance_checker/tests/test_cli.py b/compliance_checker/tests/test_cli.py
index ae7e644..e591124 100644
--- a/compliance_checker/tests/test_cli.py
+++ b/compliance_checker/tests/test_cli.py
@@ -71,3 +71,23 @@ class TestCLI(TestCase):
         )
 
         assert os.stat(path).st_size > 0
+
+    def test_json_output(self):
+        '''
+        Tests that the CF checker can produce JSON output
+        '''
+
+        fd, path = tempfile.mkstemp()
+        os.close(fd)
+        self.addCleanup(os.remove, path)
+
+        return_value, errors = ComplianceChecker.run_checker(
+            ds_loc=STATIC_FILES['conv_bad'],
+            verbose=0,
+            criteria='strict',
+            checker_names=['cf'],
+            output_filename=path,
+            output_format='json'
+        )
+
+        assert os.stat(path).st_size > 0
Repository: ioos/compliance-checker
Issue: Question about data type for flag_masks

Question about the data type allowed for a flag_masks variable.

See CF.py code 

``` python
        type_ok = v.dtype in [np.character,
                                      np.dtype('b'),
                                      np.dtype('i4'),
                                      np.int32]
```

https://github.com/ioos/compliance-checker/blob/master/compliance_checker/cf/cf.py#L811

is there a reason why int8 and int32 are allowed and int16 is not. I previously had a file
with a flag_masks variable of "short' type not passing the CF checker

Thanks

@mhidas FYI


Changed files:
compliance_checker/cf/cf.py

diff --git a/compliance_checker/cf/cf.py b/compliance_checker/cf/cf.py
index acc426f..885bd8f 100644
--- a/compliance_checker/cf/cf.py
+++ b/compliance_checker/cf/cf.py
@@ -854,11 +854,9 @@ class CFBaseCheck(BaseCheck):
                 ok_count = 0
 
                 same_type = flag_masks.dtype == v.dtype
-                type_ok = v.dtype in [np.character,
-                                      np.dtype('b'),
-                                      np.dtype('i4'),
-                                      np.int32]
-
+                type_ok = (np.issubdtype(v.dtype, int) or
+                           np.issubdtype(v.dtype, 'S') or
+                           np.issubdtype(v.dtype, 'b'))
                 if same_type:
                     ok_count += 1
                 else:
Repository: ioos/compliance-checker
Issue: Incorrect attribute name in CF test check_data_types

Whenever I run the CF checker, I get the following message at the start of the results:

```
The following exceptions occured during the cf checker (possibly indicate compliance checker issues):
cf.check_data_types: NetCDF: Attribute not found
```

Running it in verbose mode also prints a stack trace which points to line 271 of cf.py:

```
    for k, v in ds.dataset.variables.iteritems():
        if v.datatype not in [np.character,
                             ...
```

Assuming v is a NetCDF4.Variable object, it should be `v.dtype` instead of `v.datatype`.


Changed files:
compliance_checker/cf/cf.py

diff --git a/compliance_checker/cf/cf.py b/compliance_checker/cf/cf.py
index a8e805e..8f1a629 100644
--- a/compliance_checker/cf/cf.py
+++ b/compliance_checker/cf/cf.py
@@ -268,7 +268,7 @@ class CFBaseCheck(BaseCheck):
         total = len(ds.dataset.variables)
 
         for k, v in ds.dataset.variables.iteritems():
-            if v.datatype not in [np.character,
+            if v.dtype not in [np.character,
                                   np.dtype('c'),
                                   np.dtype('b'),
                                   np.dtype('i4'),