internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: opensearch-project/opensearch-build

Issue:
Make plugin integtest.sh run against non-snapshot build
The plugin integtest.sh picks up the opensearch version provided in build.gradle, which is 1.1.0-SNAPSHOT. Since the release candidates are non snapshot built artifacts, make this configurable in integ test job


Discussion:
I believe the plan is to delete all integtest.sh from all repos and have a default one in this repo. When we re-add any custom integtest.sh in repos let’s put them in a scripts/ sub folder.

Resolving patch:
diff --git a/bundle-workflow/src/paths/script_finder.py b/bundle-workflow/src/paths/script_finder.py
--- a/bundle-workflow/src/paths/script_finder.py
+++ b/bundle-workflow/src/paths/script_finder.py
@@ -64,8 +64,9 @@ def find_build_script(cls, component_name, git_dir):
     @classmethod
     def find_integ_test_script(cls, component_name, git_dir):
         paths = [
-            os.path.realpath(os.path.join(git_dir, "integtest.sh")),
-            os.path.realpath(os.path.join(git_dir, "scripts/integtest.sh")),
+            # TODO: Uncomment this after the integtest.sh tool is removed from plugin repos. See issue #497
+            # os.path.realpath(os.path.join(git_dir, "integtest.sh")),
+            # os.path.realpath(os.path.join(git_dir, "scripts/integtest.sh")),
             os.path.realpath(
                 os.path.join(cls.component_scripts_path, component_name, "integtest.sh")
             ),


Test patch:
diff --git a/bundle-workflow/tests/tests_paths/test_script_finder.py b/bundle-workflow/tests/tests_paths/test_script_finder.py
--- a/bundle-workflow/tests/tests_paths/test_script_finder.py
+++ b/bundle-workflow/tests/tests_paths/test_script_finder.py
@@ -92,6 +92,7 @@ def test_find_integ_test_script_component_override(self):
             msg="A component without scripts resolves to a component override.",
         )
 
+    @unittest.skip('See https://github.com/opensearch-project/opensearch-build/issues/497')
     def test_find_integ_test_script_component_script(self):
         self.assertEqual(
             os.path.join(self.component_with_scripts, "integtest.sh"),
@@ -101,6 +102,7 @@ def test_find_integ_test_script_component_script(self):
             msg="A component with a script resolves to the script at the root.",
         )
 
+    @unittest.skip('See https://github.com/opensearch-project/opensearch-build/issues/497')
     def test_find_integ_test_script_component_script_in_folder(self):
         self.assertEqual(
             os.path.join(self.component_with_scripts_folder, "scripts/integtest.sh"),
Repository: opensearch-project/opensearch-build

Issue:
Rename "bundle" to "distribution"?
**Is your feature request related to a problem? Please describe.**
We've been calling our output a bundle, but it's really a distribution.

**Describe the solution you'd like**
Rename bundle to distribution everywhere.



Discussion:
@julesg WDYT?
Yes, let's do this in as many places as possible. "Distribution" is the way the project describes collections of components that are built, tested, and downloaded together (see https://opensearch.org/downloads.html).

Using a different term ("bundle") for the distributions while they're being built and tested can be confusing ("At X stage in the process, is it called a bundle or a distribution? What are the rules that describe when to use each word?"), and creates different language used by 'insiders' and 'outsiders' to describe the same thing. We should always be moving in the direction of eliminating distinctions between 'insiders' and 'outsiders'. It should be simple for anybody who uses an OpenSearch distribution to find the code used to build that distribution and weigh in on it or contribute to it, so I think consistent language matters here. It's a small thing, but being welcoming requires removing both large and small barriers.

It looks like both terms evolved independently. Now that it's becoming clearer they have the same referent, I think it's worth doing some work to standardize.
Seems like @zelinh already applied some renaming, need him to check through-fully how many more renames are needed.
After checking comprehensively, this renaming from `bundle` to `dist` has been completed everywhere in bundle build file path. 

Resolving patch:
diff --git a/src/assemble_workflow/bundle_recorder.py b/src/assemble_workflow/bundle_recorder.py
--- a/src/assemble_workflow/bundle_recorder.py
+++ b/src/assemble_workflow/bundle_recorder.py
@@ -46,13 +46,13 @@ def __get_location(self, folder_name, rel_path, abs_path):
             return self.__get_public_url_path(folder_name, rel_path)
         return abs_path
 
-    # Assembled bundles are expected to be served from a separate "bundles" folder
-    # Example: https://artifacts.opensearch.org/bundles/1.0.0/<build-id
+    # Assembled output are expected to be served from a separate "dist" folder
+    # Example: https://ci.opensearch.org/ci/dbc/bundle-build/1.2.0/build-id/linux/x64/dist/
     def __get_package_location(self):
         return self.__get_location("dist", self.package_name, os.path.join(self.output_dir, self.package_name))
 
     # Build artifacts are expected to be served from a "builds" folder
-    # Example: https://artifacts.opensearch.org/builds/1.0.0/<build-id>
+    # Example: https://ci.opensearch.org/ci/dbc/bundle-build/1.2.0/build-id/linux/x64/builds/
     def __get_component_location(self, component_rel_path):
         abs_path = os.path.join(self.artifacts_dir, component_rel_path)
         return self.__get_location("builds", component_rel_path, abs_path)


Test patch:
Repository: ioos/compliance-checker

Issue:
CF check_data_types fails for certain char types
Consider the following excerpt:

``` bash
$  ncdump -v Equirectangular sss_rc201401.v3.0cap.nc
...
    char Equirectangular ;
        Equirectangular:grid_mapping_name = "latitude_longitude" ;
        Equirectangular:Standard_Parallel = 0.f ;
        Equirectangular:Longitude_of_Central_Meridian = 0.f ;
        Equirectangular:false_northing = 0.f ;
        Equirectangular:false_easting = 0.f ;
        Equirectangular:comment = "projection also referred to as Equidistant Cylindrical" ;
...
Equirectangular = "" ;
```

``` bash
data_types                             :3:     3/ 4 : [u'The variable
                                                      Equirectangular failed
                                                      because the datatype is
                                                      |S1']
```

`|S1` should be an homogenous array of single characters,  i.e. equivalent to `numpy.character`. This equivalence is further suggested by [this example from the numpy documentation](http://docs.scipy.org/doc/numpy/reference/generated/numpy.typename.html#numpy.typename).

A possible solution would be to add `|S1` to the list of allowed types in [cf.py](https://github.com/ioos/compliance-checker/blob/a758e165ba4a08a336a523a8da73dc68264897ee/compliance_checker/cf/cf.py#L277).



Discussion:
We are already allowing the data type numpy.character .  If there is an equivalence, shouldn't the S1 have passed?

@DanielJMaher, that's what I would expect! I think that they aren't _exactly_ equivalent in numpy terms, but that they do seem to map to the same netCDF4 type, `NC_CHAR` as suggested by the [netCDF4-python documentation](http://unidata.github.io/netcdf4-python/netCDF4.Dataset-class.html#createVariable).

Would you mind sending me that data file so I can test against a fix?  DMaher@ASAScience.com 

Just sent it.


Resolving patch:
diff --git a/compliance_checker/cf/cf.py b/compliance_checker/cf/cf.py
--- a/compliance_checker/cf/cf.py
+++ b/compliance_checker/cf/cf.py
@@ -283,6 +283,7 @@ def check_data_types(self, ds):
 
         for k, v in ds.dataset.variables.iteritems():
             if v.datatype not in [np.character,
+                                  np.dtype('c'),
                                   np.dtype('b'),
                                   np.dtype('i4'),
                                   np.int32,


Test patch:
Repository: ioos/compliance-checker

Issue:
Valid_coordinates not functioning properly
When a non Lat,Lon,Time,Elevation dimension is given as a coordinate, and it is not a coordinate varaible, an error was being raised.  That case should pass this check as only lat,lon,time,elev dimensions are checked here.   



Discussion:
Is this done and/or related to #20 ?

This block of code:

```
                for dim in var.dimensions:
                    if dim not in _possibleaxis:
                        break
                    elif dim not in ds.dataset.variables:
```

is checking to see if the dimension name is what we expect.  In actuality it should be the units (For lat/lon especially).  There may be more needed here.


Resolving patch:
diff --git a/compliance_checker/cf/cf.py b/compliance_checker/cf/cf.py
--- a/compliance_checker/cf/cf.py
+++ b/compliance_checker/cf/cf.py
@@ -64,6 +64,47 @@
 
 _possibleaxis = _possiblet + _possiblez + _possiblex + _possibley
 
+
+_possiblexunits = ['degrees_east',
+                    'degree_east',
+                    'degrees_E',
+                    'degree_E',
+                    'degreesE',
+                    'degreeE'
+                    ]
+
+_possibleyunits = ['degrees_north',
+                'degree_north',
+                'degrees_N',
+                'degree_N',
+                'degreesN',
+                'degreeN'
+                    ]
+
+_possibletunits = ['day', 
+                'days', 
+                'd', 
+                'hour', 
+                'hours', 
+                'hr', 
+                'hrs', 
+                'h', 
+                'year', 
+                'years', 
+                'minute', 
+                'minutes', 
+                'm', 
+                'min', 
+                'mins', 
+                'second', 
+                'seconds', 
+                's', 
+                'sec', 
+                'secs'
+                ]
+
+_possibleaxisunits =  _possiblexunits + _possibleyunits +_possibletunits
+
 def guess_dim_type(dimension):
     """
     Guesses the type of dimension of a variable X/Y/Z/T
@@ -1376,7 +1417,8 @@ def check_independent_axis_dimensions(self, ds):
                 reasoning = []
                 valid_coordinate = True
                 for dim in var.dimensions:
-                    if dim not in _possibleaxis:
+                    #Looks to see if the dimension name or unit is not in the alloweable lists.  
+                    if dim not in _possibleaxis or dim.units not in _possibleaxisunits or dim.units.split(" ")[0] not in _possibleaxisunits or hasattr(dim,'positive'):
                         break
                     elif dim not in ds.dataset.variables:
                         valid_coordinate = False


Test patch:
Repository: ioos/compliance-checker

Issue:
Cell Boundaries
The Cell Boundaries Check is not properly finding the length of the variable and the length of the cell boundary variable.  This is causing inaccurate failures.  



Discussion:

Resolving patch:
diff --git a/compliance_checker/cf/cf.py b/compliance_checker/cf/cf.py
--- a/compliance_checker/cf/cf.py
+++ b/compliance_checker/cf/cf.py
@@ -1882,37 +1882,18 @@ def check_cell_boundaries(self, ds):
         reasoning = []
         valid = ' '
 
-        for name, var in ds.dataset.variables.iteritems():
-            for dim in var.dimensions:
-                if dim == name:
-                    if getattr(var, 'bounds', ''):
-                        bounds = getattr(var,'bounds','')
-
-
-                        if len(ds.dataset.variables[bounds].shape) == len(var.shape):
-                            last_shape = ds.dataset.variables[bounds].shape[-1]
-                        elif len(ds.dataset.variables[bounds].shape) == len(var.shape)+1:
-                            last_shape = 1
-
-
-                        if ds.dataset.variables[bounds].shape[-1]  == last_shape + 1:
-                            valid = True
-                            reasoning.append('The shape of the Coordinate Variable is M x N Boundary Variable is M x N+1.')
-
-                        elif ds.dataset.variables[name].shape == var.shape[0:-1] and last_shape == 1:
-                            valid = True
-                            reasoning.append('The shape of the Coordinate Variable is M x N Boundary Variable is M x N x 1.')
-
-                        else:
-                            valid = False
-                            reasoning.append('The shape of the Boundary Variable is wrong.')
-                        result = Result(BaseCheck.MEDIUM,                            \
-                                    valid,                                       \
-                                    ('var', name, 'cell_boundaries'), \
-                                    reasoning)
-                        ret_val.append(result)
-                        reasoning = []
+        for cvar, bvar in self._find_boundary_vars(ds).iteritems():
+                valid = True
+                if bvar.ndim !=cvar.ndim + 1:
+                    valid = False
+                    reasoning.append('The number of dimensions of the Coordinate Variable is %s, but the number of dimensions of the Boundary Variable is %s.'%(cvar.ndim, bvar.ndim))
 
+                result = Result(BaseCheck.MEDIUM,                          
+                            valid,                                       
+                            ('var', cvar._name, 'cell_boundaries'), 
+                            reasoning)
+                ret_val.append(result)
+                reasoning = []
 
         return ret_val
 


Test patch:
Repository: ioos/compliance-checker

Issue:
Return value always 0
In order to incorporate compliance-checker efficiently with other command line tools, it is recommended that it returns a non-zero return value on error.

At the moment it returns 0 all the time, correct me if I'm wrong.

Here is a sample output:

```
$ ./cchecker.py -t tester -c strict /tmp/output.nc
Running Compliance Checker on the dataset from: /tmp/output.nc


--------------------------------------------------------------------------------
                      The dataset scored 0 out of 2 points                      
                              during the tester check                              
--------------------------------------------------------------------------------
                               Scoring Breakdown:                               


                                 High Priority                                  
--------------------------------------------------------------------------------
    Name                            :Priority: Score
fail_me                                 :3:     0/1
fail_me2                                :3:     0/1
                        No Medium priority tests present                        
--------------------------------------------------------------------------------

                         No Low priority tests present                          
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------
                  Reasoning for the failed tests given below:                   


Name                             Priority:     Score:Reasoning
--------------------------------------------------------------------------------
fail_me                                :3:     0/ 1 : Attr fail_me not present
fail_me2                               :3:     0/ 1 : Attr fail_me2 not present
$ echo $?
0
```



Discussion:
This is indeed a regression from pre-1.0 code - it was always supposed to exit with a code.


Resolving patch:
diff --git a/cchecker.py b/cchecker.py
--- a/cchecker.py
+++ b/cchecker.py
@@ -1,7 +1,7 @@
 #!/usr/bin/env python
 import pprint
 import argparse
-
+import sys
 from compliance_checker.runner import ComplianceChecker, ComplianceCheckerCheckSuite
 
 def main():
@@ -15,10 +15,13 @@ def main():
 
     print "Running Compliance Checker on the dataset from: %s" % args.dataset_location[0]
 
-    ComplianceChecker.run_checker(args.dataset_location[0],
+    return_value = ComplianceChecker.run_checker(args.dataset_location[0],
                                   args.test,
                                   args.verbose,
                                   args.criteria)
+    if return_value is not True:
+        return 1
+    return 0
 
 if __name__ == "__main__":
-    main()
+    sys.exit(main())


Test patch:
Repository: ioos/compliance-checker

Issue:
Incorrect attribute name in CF test check_data_types
Whenever I run the CF checker, I get the following message at the start of the results:

```
The following exceptions occured during the cf checker (possibly indicate compliance checker issues):
cf.check_data_types: NetCDF: Attribute not found
```

Running it in verbose mode also prints a stack trace which points to line 271 of cf.py:

```
    for k, v in ds.dataset.variables.iteritems():
        if v.datatype not in [np.character,
                             ...
```

Assuming v is a NetCDF4.Variable object, it should be `v.dtype` instead of `v.datatype`.



Discussion:

Resolving patch:
diff --git a/compliance_checker/cf/cf.py b/compliance_checker/cf/cf.py
--- a/compliance_checker/cf/cf.py
+++ b/compliance_checker/cf/cf.py
@@ -268,7 +268,7 @@ def check_data_types(self, ds):
         total = len(ds.dataset.variables)
 
         for k, v in ds.dataset.variables.iteritems():
-            if v.datatype not in [np.character,
+            if v.dtype not in [np.character,
                                   np.dtype('c'),
                                   np.dtype('b'),
                                   np.dtype('i4'),


Test patch:
Repository: ioos/compliance-checker

Issue:
Do ancillary variables really need a coordinates attribute?
For discrete sampling geometry (DSG) files, the function `check_coordinates_and_metadata` (in cf.py) returns a fail for variables that don't have a **coordinates** attribute. It exempts variables that are themselves coordinate variables, but not other types of ancillary variables. So for examples we get fails for all our quality control variables.

Could we exclude any variable listed in another's **ancillary_variables** attribute?



Discussion:

Resolving patch:
diff --git a/compliance_checker/cf/cf.py b/compliance_checker/cf/cf.py
--- a/compliance_checker/cf/cf.py
+++ b/compliance_checker/cf/cf.py
@@ -2738,8 +2738,11 @@ def check_coordinates_and_metadata(self, ds):
                     if each in name_list:
                         non_data_list.append(each)
 
-        data_list = [each for each in name_list if each not in non_data_list]
+            if hasattr(var, 'ancillary_variables'):
+                for each in getattr(var, 'ancillary_variables', '').split(' '):
+                    non_data_list.append(each)
 
+        data_list = [each for each in name_list if each not in non_data_list]
 
         for each in data_list:
             if getattr(ds.dataset.variables[each], 'coordinates', ''):


Test patch:
diff --git a/compliance_checker/tests/data/coordinates_and_metadata.nc b/compliance_checker/tests/data/coordinates_and_metadata.nc
new file mode 100644
Binary files /dev/null and b/compliance_checker/tests/data/coordinates_and_metadata.nc differ
diff --git a/compliance_checker/tests/test_cf.py b/compliance_checker/tests/test_cf.py
--- a/compliance_checker/tests/test_cf.py
+++ b/compliance_checker/tests/test_cf.py
@@ -33,7 +33,8 @@
         'index_ragged'         : resource_filename('compliance_checker', 'tests/data/index_ragged.nc'),
         'bad_missing_data'     : resource_filename('compliance_checker', 'tests/data/bad_missing_data.nc'),
         'self-referencing-var' : resource_filename('compliance_checker', 'tests/data/self-referencing-var.nc'),
-        'scalar_coordinate_variable' : resource_filename('compliance_checker', 'tests/data/scalar_coordinate_variable.nc')
+        'scalar_coordinate_variable' : resource_filename('compliance_checker', 'tests/data/scalar_coordinate_variable.nc'),
+        'coordinates_and_metadata' : resource_filename('compliance_checker', 'tests/data/coordinates_and_metadata.nc')
         }
 
 class MockVariable(object):
@@ -726,6 +727,12 @@ def test_check_coordinates_and_metadata(self):
         results = self.cf.check_coordinates_and_metadata(dataset)
         self.assertTrue(results[-1].value)
 
+        dataset = self.get_pair(static_files['coordinates_and_metadata'])
+        results = self.cf.check_coordinates_and_metadata(dataset)
+        self.assertTrue(len(results) == 2)
+        self.assertFalse(results[0].value)
+        self.assertFalse(results[1].value)
+
     def test_check_missing_data(self):
         dataset = self.get_pair(static_files['index_ragged'])
         results = self.cf.check_missing_data(dataset)
Repository: ioos/compliance-checker

Issue:
CF check_units should skip "feature id" variables
A variable with the attribute _cf_role_ is used to uniquely identify each feature in a discrete-sampling-geometry (DSG) file ([CF ref](http://cfconventions.org/Data/cf-conventions/cf-conventions-1.6/build/cf-conventions.html#coordinates-metadata)). It can have any type, but even if it's numeric, it may not make sense for it to have units (e.g. it could be a serial number for profiles taken on a cruise).

May I propose that any such variable be skipped in check_units (cf.py)?



Discussion:
For example [this file](http://thredds.aodn.org.au/thredds/catalog/IMOS/eMII/checker_test/ANMN/profile/catalog.html?dataset=IMOS/eMII/checker_test/ANMN/profile/IMOS_ANMN-NRS_CDEKOSTUZ_20141216T020100Z_NRSROT_FV01_3080.0-1-SBE19plus-45_C-20141219T063744Z.fixed.nc) has a variable:

```
    int INSTANCE(INSTANCE) ;
            INSTANCE:name = "INSTANCE" ;
            INSTANCE:long_name = "instance_of_vertical_profiles_in_data_file" ;
            INSTANCE:cf_role = "profile_id" ;
```

and currently results in this fail result:

```
units                                  :3:    12/13 :  
    INSTANCE                           :3:     0/ 1 :  
        present                        :3:     0/ 1 : units attribute required
```


Resolving patch:
diff --git a/compliance_checker/cf/cf.py b/compliance_checker/cf/cf.py
--- a/compliance_checker/cf/cf.py
+++ b/compliance_checker/cf/cf.py
@@ -499,6 +499,10 @@ def check_units(self, ds):
             
             if hasattr(v, 'standard_name') and 'status_flag' in v.standard_name:
                 continue
+
+            # skip DSG cf_role
+            if hasattr(v, "cf_role"):
+                continue
             
             units = getattr(v, 'units', None)
 


Test patch: