anon-iclr-submission/benchname-bug-localization huggingface dataset:
Repository: esmvalgroup/esmvalcore
Issue: ESMValTool crashes when recipe uses ancestors

**Describe the bug**
The tool crashes with this error when I run a recipe which uses the `ancestors` key, other recipes run fine (I tested the python and NCL example):

```
Traceback (most recent call last):
  File "/miniconda3/envs/mlr/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/ESMValCore/esmvalcore/_task.py", line 684, in _run_task
    output_files = task.run()
  File "/ESMValCore/esmvalcore/_task.py", line 238, in run
    self.output_files = self._run(input_files)
  File "/ESMValCore/esmvalcore/_task.py", line 454, in _run
    f for f in input_files
  File "/ESMValCore/esmvalcore/_task.py", line 455, in <listcomp>
    if f.endswith('.yml') or os.path.isdir(f)
AttributeError: 'PosixPath' object has no attribute 'endswith'
```

Example: [`ESMValTool/esmvaltool/recipes/examples/recipe_python_object_oriented.yml`](https://github.com/ESMValGroup/ESMValTool/blob/master/esmvaltool/recipes/examples/recipe_python_object_oriented.yml).

Probably related to #528.

(Using the most recent `master` branches of ESMValTool and ESMValCore)

Changed files:
esmvalcore/_task.py

diff --git a/esmvalcore/_task.py b/esmvalcore/_task.py
index c20aebeb4..fe5136c7e 100644
--- a/esmvalcore/_task.py
+++ b/esmvalcore/_task.py
@@ -268,7 +268,7 @@ class DiagnosticTask(BaseTask):
         super().__init__(ancestors=ancestors, name=name)
         self.script = script
         self.settings = settings
-        self.output_dir = Path(output_dir)
+        self.output_dir = output_dir
         self.cmd = self._initialize_cmd()
         self.log = Path(settings['run_dir']) / 'log.txt'
         self.resource_log = Path(settings['run_dir']) / 'resource_usage.txt'
Repository: esmvalgroup/esmvalcore
Issue: `check.py` still calls coord.bounds() instead of coord.core_bounds() in some cases

**Describe the bug**
In #1912 , the cmor checker calls to the coordinates was made to use the array module specific to the coordinate, instead of automatically converting everything to numpy arrays.

There is still one case in which the coordinate bounds for an aux coord (3D array) may get realised:

https://github.com/ESMValGroup/ESMValCore/blob/f00b6c6190a114ffe12aea007bd31b066d2cb8d9/esmvalcore/cmor/check.py#L776

It would be good to change that case for it to call coord.core_bounds()

Changed files:
esmvalcore/cmor/check.py

diff --git a/esmvalcore/cmor/check.py b/esmvalcore/cmor/check.py
index 29f0f74e7..4eb057730 100644
--- a/esmvalcore/cmor/check.py
+++ b/esmvalcore/cmor/check.py
@@ -773,8 +773,8 @@ class CMORCheck():
             else:
                 new_lons = coord.core_points().copy()
                 new_lons = self._set_range_in_0_360(new_lons)
-                if coord.bounds is not None:
-                    new_bounds = coord.bounds.copy()
+                if coord.core_bounds() is not None:
+                    new_bounds = coord.core_bounds().copy()
                     new_bounds = self._set_range_in_0_360(new_bounds)
                 else:
                     new_bounds = None
Repository: esmvalgroup/esmvalcore
Issue: Bug in fix for FGOALS-g2

The fix for FGOALS-g2 (CMIP5) does not work on `fx` variable, since it attempts to fix the time coordinate, which is not defined in `fx` variables:
```
        for cube in cubes:
            time = cube.coord('time')
            time.units = Unit(time.units.name, time.units.calendar)
```

Changed files:
tests/integration/cmor/_fixes/cmip5/test_fgoals_g2.py, esmvalcore/cmor/_fixes/cmip5/fgoals_g2.py

diff --git a/esmvalcore/cmor/_fixes/cmip5/fgoals_g2.py b/esmvalcore/cmor/_fixes/cmip5/fgoals_g2.py
index 8811d3953..23c032abb 100644
--- a/esmvalcore/cmor/_fixes/cmip5/fgoals_g2.py
+++ b/esmvalcore/cmor/_fixes/cmip5/fgoals_g2.py
@@ -1,6 +1,7 @@
 
 """Fix FGOALS-g2 model."""
 from cf_units import Unit
+from iris.exceptions import CoordinateNotFoundError
 
 from ..fix import Fix
 
@@ -24,6 +25,10 @@ class AllVars(Fix):
 
         """
         for cube in cubes:
-            time = cube.coord('time')
-            time.units = Unit(time.units.name, time.units.calendar)
+            try:
+                time = cube.coord('time')
+            except CoordinateNotFoundError:
+                pass
+            else:
+                time.units = Unit(time.units.name, time.units.calendar)
         return cubes
diff --git a/tests/integration/cmor/_fixes/cmip5/test_fgoals_g2.py b/tests/integration/cmor/_fixes/cmip5/test_fgoals_g2.py
index 805a2bf1b..a17eeeb29 100644
--- a/tests/integration/cmor/_fixes/cmip5/test_fgoals_g2.py
+++ b/tests/integration/cmor/_fixes/cmip5/test_fgoals_g2.py
@@ -36,3 +36,8 @@ class TestAll(unittest.TestCase):
         self.assertEqual(time.units.origin,
                          'day since 1-01-01 00:00:00.000000')
         self.assertEqual(time.units.calendar, 'gregorian')
+
+    def test_fix_metadata_dont_gail_if_not_time(self):
+        """Test calendar fix."""
+        self.cube.remove_coord('time')
+        self.fix.fix_metadata([self.cube])
Repository: esmvalgroup/esmvalcore
Issue: Bug in activity_id determination for some cmip6 experiments

In PR #107 (addressing issue #37), we introduced the automatic determination of `activity_id` from `experiment_id`. This works nicely in most cases, but a few experiments have more than one activity associated with them. In that case, all activities are listed in the CV field separated by spaces, but following [the official DRS standard (bottom of page 17)](http://goo.gl/v1drZl), only the first one is used in the path.

As a fix, the field coming from the CV should be `split` on `" "` and only the first entry be taken. This should also work in the normal case of only one activity.

The list of afflicted experiments is
- dcppC-forecast-addPinatubo
- esm-1pct-brch-1000PgC
- esm-1pct-brch-2000PgC
- esm-1pct-brch-750PgC
- esm-1pctCO2
- esm-bell-1000PgC
- esm-bell-2000PgC
- esm-bell-750PgC
- land-hist
- piClim-aer
- piClim-control
- ssp370


Changed files:
tests/integration/test_recipe.py, esmvalcore/_config.py, esmvalcore/cmor/table.py

diff --git a/esmvalcore/_config.py b/esmvalcore/_config.py
index c3710141f..5caccda7f 100644
--- a/esmvalcore/_config.py
+++ b/esmvalcore/_config.py
@@ -190,8 +190,8 @@ def get_activity(variable):
         exp = variable['exp']
         logger.debug("Retrieving activity_id for experiment %s", exp)
         if isinstance(exp, list):
-            return [CMOR_TABLES[project].activities[value] for value in exp]
-        return CMOR_TABLES[project].activities[exp]
+            return [CMOR_TABLES[project].activities[value][0] for value in exp]
+        return CMOR_TABLES[project].activities[exp][0]
     except (KeyError, AttributeError):
         return None
 
diff --git a/esmvalcore/cmor/table.py b/esmvalcore/cmor/table.py
index f179f8193..d876f9d06 100644
--- a/esmvalcore/cmor/table.py
+++ b/esmvalcore/cmor/table.py
@@ -191,7 +191,7 @@ class CMIP6Info(object):
                 try:
                     exps = table_data['CV']['experiment_id']
                     for exp_id in exps:
-                        activity = exps[exp_id]['activity_id']
+                        activity = exps[exp_id]['activity_id'][0].split(' ')
                         self.activities[exp_id] = activity
                 except (KeyError, AttributeError):
                     pass
diff --git a/tests/integration/test_recipe.py b/tests/integration/test_recipe.py
index 31d2d7853..fa73f5ed7 100644
--- a/tests/integration/test_recipe.py
+++ b/tests/integration/test_recipe.py
@@ -408,7 +408,7 @@ def test_cmip6_variable_autocomplete(tmp_path, patched_datafinder,
     variable = recipe.diagnostics['test']['preprocessor_output']['pr'][0]
 
     reference = {
-        'activity': ['CMIP'],
+        'activity': 'CMIP',
         'dataset': 'HadGEM3-GC31-LL',
         'diagnostic': 'test',
         'end_year': 2001,
Repository: esmvalgroup/esmvalcore
Issue: Zonal preprocessor return type float64

**Describe the bug**
The zonal preprocessor returns a cube of type `float64`, instead of `float32` as for other preprocessors.
This is an issue for the NCL diagnostics, which are designed to work with variable of `float32` type (as per CMOR standard).

Hopefully it is easy to fix this.

Changed files:
esmvalcore/preprocessor/_area.py

diff --git a/esmvalcore/preprocessor/_area.py b/esmvalcore/preprocessor/_area.py
index dfb39a74a..9f023dd02 100644
--- a/esmvalcore/preprocessor/_area.py
+++ b/esmvalcore/preprocessor/_area.py
@@ -114,7 +114,9 @@ def zonal_statistics(cube, operator):
     """
     if cube.coord('longitude').points.ndim < 2:
         operation = get_iris_analysis_operation(operator)
-        return cube.collapsed('longitude', operation)
+        cube = cube.collapsed('longitude', operation)
+        cube.data = cube.core_data().astype(np.float32, casting='same_kind')
+        return cube
     else:
         msg = (f"Zonal statistics on irregular grids not yet implemnted")
         raise ValueError(msg)
@@ -146,7 +148,9 @@ def meridional_statistics(cube, operator):
     """
     if cube.coord('latitude').points.ndim < 2:
         operation = get_iris_analysis_operation(operator)
-        return cube.collapsed('latitude', operation)
+        cube = cube.collapsed('latitude', operation)
+        cube.data = cube.core_data().astype(np.float32, casting='same_kind')
+        return cube
     else:
         msg = (f"Meridional statistics on irregular grids not yet implemnted")
         raise ValueError(msg)
Repository: idaholab/civet
Issue: Bug in branch color

I noticed last night (and still this morning) that the branch color for MOOSE-devel is currently yellow even though there are no jobs running anywhere that I can see. John just pushed a libMesh PR so that PR is yellow but so is the branch color of MOOSE-devel.


Changed files:
ci/client/views.py

diff --git a/ci/client/views.py b/ci/client/views.py
index 4537fd2..fbea3f6 100644
--- a/ci/client/views.py
+++ b/ci/client/views.py
@@ -32,11 +32,11 @@ def update_status(job, status=None):
       job.event.pull_request.status = status
       job.event.pull_request.save()
   elif job.event.base.branch:
-    latest_event = models.Event.objects.filter(base__branch=job.event.base.branch).order_by('-created').first()
-    if latest_event == job.event:
-      # only update branch status if this is the latest event
-      job.event.base.branch.status = status
-      job.event.base.branch.save()
+    # always update the branch status. This differs from the pull request logic
+    # since previous events get canceled on a PR but we can have multiple events
+    # running on a branch.
+    job.event.base.branch.status = status
+    job.event.base.branch.save()
 
 def get_client_ip(request):
   x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR')
Repository: amwa-tv/nmos-testing
Issue: IS-04-01 test_21 the mock registry does not return location header

During test_21, the mock registry returns a HTTP 200 when trying to register, the node should then perform a DELETE using the location header in the response, but the header is missing from the 200 response

Changed files:
nmostesting/mocks/Registry.py

diff --git a/nmostesting/mocks/Registry.py b/nmostesting/mocks/Registry.py
index 7cc32c1..5640d4a 100644
--- a/nmostesting/mocks/Registry.py
+++ b/nmostesting/mocks/Registry.py
@@ -138,10 +138,12 @@ def post_resource(version):
     else:
         registered = True
     registry.add(request.headers, request.json, version)
+    location = "/x-nmos/registration/{}/resource/{}/{}".format(version, request.json["type"],
+                                                               request.json["data"]["id"])
     if registered:
-        return jsonify(request.json["data"]), 200
+        return jsonify(request.json["data"]), 200, {"Location": location}
     else:
-        return jsonify(request.json["data"]), 201
+        return jsonify(request.json["data"]), 201, {"Location": location}
 
 
 @REGISTRY_API.route('/x-nmos/registration/<version>/resource/<resource_type>/<resource_id>', methods=["DELETE"])
Repository: amwa-tv/nmos-testing
Issue: TypeError in IS0401Test test_13

String indices must be integers on line 359 test_13, where stream_type is identified.

```
 * Running test_13
 * ERROR: string indices must be integers
192.168.200.43 - - [15/Jan/2019 11:36:20] "POST / HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1741, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.5/dist-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.5/dist-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/ipstudio-admin/nmos-testing/nmos-test.py", line 218, in index_page
    raise ex
  File "/home/ipstudio-admin/nmos-testing/nmos-test.py", line 215, in index_page
    result = test_obj.run_tests(test_selection)
  File "/home/ipstudio-admin/nmos-testing/GenericTest.py", line 133, in run_tests
    self.execute_tests(test_name)
  File "/home/ipstudio-admin/nmos-testing/GenericTest.py", line 101, in execute_tests
    self.result.append(method())
  File "/home/ipstudio-admin/nmos-testing/IS0401Test.py", line 359, in test_13
    stream_type = receiver["format"].split(":")[-1]
TypeError: string indices must be integers

```

Changed files:
IS0401Test.py

diff --git a/IS0401Test.py b/IS0401Test.py
index af5ee5c..774e9ef 100644
--- a/IS0401Test.py
+++ b/IS0401Test.py
@@ -356,7 +356,11 @@ class IS0401Test(GenericTest):
         try:
             formats_tested = []
             for receiver in receivers.json():
-                stream_type = receiver["format"].split(":")[-1]
+                try:
+                    stream_type = receiver["format"].split(":")[-1]
+                except TypeError:
+                    return test.FAIL("Unexpected Receiver format: {}".format(receiver))
+
                 # Test each available receiver format once
                 if stream_type in formats_tested:
                     continue
Repository: amwa-tv/nmos-testing
Issue: MdnsListener has no attribute remove_service

AttributeError when using zeroconf mDNS.

```
 * Running test_13
Exception in thread zeroconf-ServiceBrowser__nmos-node._tcp.local.:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.5/dist-packages/zeroconf.py", line 1382, in run
    handler(self.zc)
  File "/usr/local/lib/python3.5/dist-packages/zeroconf.py", line 1329, in <lambda>
    state_change=state_change,
  File "/usr/local/lib/python3.5/dist-packages/zeroconf.py", line 1235, in fire
    h(**kwargs)
  File "/usr/local/lib/python3.5/dist-packages/zeroconf.py", line 1304, in on_change
    listener.remove_service(*args)
AttributeError: 'MdnsListener' object has no attribute 'remove_service'

```

Changed files:
MdnsListener.py

diff --git a/MdnsListener.py b/MdnsListener.py
index b0627a7..c977494 100644
--- a/MdnsListener.py
+++ b/MdnsListener.py
@@ -30,6 +30,9 @@ class MdnsListener(object):
         t.daemon = True
         t.start()
 
+    def remove_service(self, zeroconf, srv_type, name):
+        pass
+
     def get_service_list(self):
         self.resolve_queue.join()
         return self.services
Repository: amwa-tv/nmos-testing
Issue: Automatic tests do not support sub-resources in channel mapping spec

The channel mapping /activations resource uses a new key/value pair format, with sub-resources accessed via a GET to /activations/{key}. The test suite doesn't support this format yet.

Changed files:
GenericTest.py

diff --git a/GenericTest.py b/GenericTest.py
index d713b8e..899fd38 100644
--- a/GenericTest.py
+++ b/GenericTest.py
@@ -318,6 +318,11 @@ class GenericTest(object):
                     elif isinstance(entry, str) and entry.endswith("/"):
                         res_id = entry.rstrip("/")
                         subresources.append(res_id)
+            elif isinstance(response.json(), dict):
+                for key, value in response.json().items():
+                    # Cover the audio channel mapping spec case with dictionary keys
+                    if isinstance(key, str) and isinstance(value, dict):
+                        subresources.append(key)
         except json.decoder.JSONDecodeError:
             pass
Repository: amwa-tv/nmos-testing
Issue: Git clone fails if a remote branch has been deleted

When a v1.x-dev branch gets deleted, if you previously had it in your local cache of the spec files, the next git pull will fail. The only solution to this at present is to manually delete the cache directory. Ideally this should be detected and handled more gracefully.

Changed files:
nmos-test.py

diff --git a/nmos-test.py b/nmos-test.py
index ad08261..657688b 100644
--- a/nmos-test.py
+++ b/nmos-test.py
@@ -259,8 +259,12 @@ if __name__ == '__main__':
             # Only pull if we haven't in the last hour
             if (last_pull_time + timedelta(hours=1)) <= time_now:
                 print(" * Pulling latest files for repository '{}'".format(repo_data["repo"]))
-                repo.remotes.origin.pull()
-                update_last_pull = True
+                try:
+                    repo.remotes.origin.pull()
+                    update_last_pull = True
+                except Exception as e:
+                    print(" * ERROR: Unable to update repository '{}'. If the problem persists, "
+                          "please delete the '{}' directory".format(repo_data["repo"], CACHE_PATH))
 
     if update_last_pull:
         try:
Repository: amwa-tv/nmos-testing
Issue: Ramlfications fails to parse IS-07 global types

The IS-07 spec appears to use valid RAML 1.0, but ramlfications (being a 0.8 library) can't quite handle the way they're written. As such several tests appear as 'manual' when they should work.

Changed files:
Specification.py

diff --git a/Specification.py b/Specification.py
index 06d1c0e..fbdd3b8 100644
--- a/Specification.py
+++ b/Specification.py
@@ -60,19 +60,40 @@ class Specification(object):
         """Fixes RAML files to match ramlfications expectations (bugs)"""
         lines = []
         in_schemas = False
+        type_name = None
         try:
             with open(file_path) as raml:
+                # Read in a single line of RAML
                 line = raml.readline()
                 while line:
                     if in_schemas and not line.startswith(" "):
+                        # Detect that we've reached the first line after the schemas/types section
                         in_schemas = False
-                    if in_schemas and "- " not in line:
+
+                    if in_schemas and "!include" not in line:
+                        # This is a correct RAML 1.0 type definition
+                        type_name = line
+                    elif in_schemas and type_name is not None:
+                        # Make the RAML 1.0 type definition look more like RAML 0.8 to aid parsing
+                        line = "  - " + type_name.strip() + " " + line.replace("type:", "").lstrip()
+                        type_name = None
+                    elif in_schemas and "- " not in line:
+                        # Add a leading dash to type/schema definitions to ensure they can be read by ramlfications
                         line = "  - " + line.lstrip()
+
                     if line.startswith("schemas:") or line.startswith("types:"):
+                        # We've hit the global schema/type definition section
                         in_schemas = True
+
                     if line.startswith("traits:"):
-                        line = "bugfix:\\r\\n"  # Work around issue with ramlfications utils.py '_remove_duplicates'
-                    lines.append(line)
+                        # Remove traits to work around an issue with ramlfications util.py '_remove_duplicates'
+                        line = "bugfix:\\r\\n"
+
+                    if type_name is None:
+                        # Assuming we're not in the middle of fixing a RAML 1.0 type def, add the line to the output RAML
+                        lines.append(line)
+
+                    # Read the next line of the RAML file
                     line = raml.readline()
             with open(file_path, "w") as raml:
                 raml.writelines("".join(lines))
Repository: amwa-tv/nmos-testing
Issue: IS-08 test 5 incorrectly handles 204 responses

Due to an earlier change. The 'string' parameter no longer exists, so a 204 code with an empty body is attempted to be parsed as JSON.

https://github.com/AMWA-TV/nmos-testing/blob/master/is08/activation.py#L97

Changed files:
nmostesting/suites/is08/calls.py

diff --git a/nmostesting/suites/is08/calls.py b/nmostesting/suites/is08/calls.py
index 9efa195..782c46d 100644
--- a/nmostesting/suites/is08/calls.py
+++ b/nmostesting/suites/is08/calls.py
@@ -52,7 +52,10 @@ class Call:
         self._checkForErrors()
         self._checkStatusCode()
         self._checkResponseSchema()
-        return self._getJSON()
+        if self.expectedCode != 204:
+            return self._getJSON()
+        else:
+            return None
 
     def _checkForErrors(self):
         if not self._callSucceeded:
Repository: amwa-tv/nmos-testing
Issue: check_matching_resource checking the wrong thing?

As of commit c18cc3f357be9deb81f85ff35378519fb2498aec, nmos-cpp-node from the sony/nmos-cpp project started failing several unit tests, starting with test_04 ("Node can register a valid Node..."), with the message "Node API JSON does not match data in registry...". Investigating further, it appears that left-hand side of the comparison has a whole lot more data than the right-hand side and that limiting the data being compared make the tests succeed. In particular, if I change line 195 from `elif reg_resource != node_resources[resource]:` to `elif reg_resource[1]['payload']['data'] != node_resources[resource]:`, the tests pass.

Does that seem like a reasonable fix, or is there something more subtle going on here?

[edit: my original proposed fix would break the test at line 193]

Changed files:
IS0401Test.py

diff --git a/IS0401Test.py b/IS0401Test.py
index 9fd3549..606c7ed 100644
--- a/IS0401Test.py
+++ b/IS0401Test.py
@@ -148,7 +148,7 @@ class IS0401Test(GenericTest):
             # Look up data in local mock registry
             for resource in self.registry.get_data():
                 if resource[1]["payload"]["type"] == res_type and resource[1]["payload"]["data"]["id"] == res_id:
-                    found_resource = resource
+                    found_resource = resource[1]["payload"]["data"]
         else:
             # Look up data from a configured Query API
             url = "http://" + QUERY_API_HOST + ":" + str(QUERY_API_PORT) + "/x-nmos/query/" + \\
@@ -188,13 +188,13 @@ class IS0401Test(GenericTest):
                     if len(node_resources) == 0:
                         return test.NA("No {} resources were found on the Node.".format(res_type.title()))
 
-                    for resource in node_resources:
-                        reg_resource = self.get_registry_resource(res_type, resource)
+                    for res_id in node_resources:
+                        reg_resource = self.get_registry_resource(res_type, res_id)
                         if not reg_resource:
-                            return test.FAIL("{} {} was not found in the registry.".format(res_type.title(), resource))
-                        elif reg_resource != node_resources[resource]:
+                            return test.FAIL("{} {} was not found in the registry.".format(res_type.title(), res_id))
+                        elif reg_resource != node_resources[res_id]:
                             return test.FAIL("Node API JSON does not match data in registry for "
-                                             "{} {}.".format(res_type.title(), resource))
+                                             "{} {}.".format(res_type.title(), res_id))
 
                     return test.PASS()
                 except ValueError:
Repository: amwa-tv/nmos-testing
Issue: IS-08-01: test_15, uncaught exception: 'list' object has no attribute 'id'

test_15 produces an exception due to `output` variable being a list instead of being just an output object. 
```
def test_15(self):
...
 chosenInput = constrainedInputs[0]
        output = chosenInput.getRoutableOutputs()
        action = Action(
            chosenInput.id,
            output.id
        )
...
```
This can be fixed by accessing the first element of `output` in the `action` construction so that `output.id` becomes `output[0].id`.


Changed files:
nmostesting/suites/IS0801Test.py

diff --git a/nmostesting/suites/IS0801Test.py b/nmostesting/suites/IS0801Test.py
index 8c087fb..938e874 100644
--- a/nmostesting/suites/IS0801Test.py
+++ b/nmostesting/suites/IS0801Test.py
@@ -367,7 +367,7 @@ class IS0801Test(GenericTest):
         output = chosenInput.getRoutableOutputs()
         action = Action(
             chosenInput.id,
-            output.id
+            output[0].id
         )
         activation = Activation()
         activation.addAction(action)
Repository: amwa-tv/nmos-testing
Issue: IS-08 Audio Channel Mapping API: AUTO test requests /map/active/activation

I had not yet the time to look into the generation of the auto tests in detail, but it looks like there happens something weird.

Running the tests gives me:
```
[...]
auto_17 | Fail | GET /x-nmos/channelmapping/v1.0/map/active/activation | Incorrect response code: 404 
[...]
```
Looks like the `save_subresources` needs an update to handle the `active/` endpoint correctly, which has a 
```{
    "activation": {
        "activation_time": "1554725653:948383693",
        "mode": "activate_scheduled_relative",
        "requested_time": "5:0"
    },
    "map": {
        "501ed417-0309-4b82-865a-f6fa5406f451": {
            "0": {
                "channel_index": 0,
                "input": "f93fda5d-3f60-49b0-8fec-91ace761c888"
            },
            "1": {
                "channel_index": 0,
                "input": "f93fda5d-3f60-49b0-8fec-91ace761c888"
            }
            [...]
```


Changed files:
IS0801Test.py, IS0802Test.py

diff --git a/IS0801Test.py b/IS0801Test.py
index 9178d6f..3bdc387 100644
--- a/IS0801Test.py
+++ b/IS0801Test.py
@@ -34,7 +34,11 @@ class IS0801Test(GenericTest):
     """
 
     def __init__(self, apis):
-        GenericTest.__init__(self, apis)
+        # Don't auto-test /map/active/{outputId} as the tests cannot find the {outputId}s automatically
+        omit_paths = [
+            "/map/active/{outputId}"
+        ]
+        GenericTest.__init__(self, apis, omit_paths)
         globalConfig.apiUrl = apis[MAPPING_API_KEY]['url']
         globalConfig.testSuite = self
         globalConfig.apiKey = MAPPING_API_KEY
diff --git a/IS0802Test.py b/IS0802Test.py
index 05dec4d..67f7e51 100644
--- a/IS0802Test.py
+++ b/IS0802Test.py
@@ -31,7 +31,11 @@ class IS0802Test(GenericTest):
     """
 
     def __init__(self, apis):
-        GenericTest.__init__(self, apis)
+        # Don't auto-test /map/active/{outputId} as the tests cannot find the {outputId}s automatically
+        omit_paths = [
+            "/map/active/{outputId}"
+        ]
+        GenericTest.__init__(self, apis, omit_paths)
         globalConfig.apiUrl = apis[MAPPING_API_KEY]['url']
         globalConfig.testSuite = self
         globalConfig.apiKey = MAPPING_API_KEY
Repository: amwa-tv/nmos-testing
Issue: IS-08-01: test_13 will fail when test_06 is run right before it

This issue arises when executing all the `IS-08-01` testcases together. `test_06` executes a relative activation in 5 seconds and expects a `423 response` due to a locked output. However the activation is not cleared before running the next testcases causing `test_13` to fail since it will also produce a `423 response`.

It can be solved by deleting the activation after checking the lock in `test_06`:
```
    def test_06(self, test):
        """Attempting to change a locked route results in a 423 response"""
        globalConfig.test = test

        outputList = getOutputList()
        if len(outputList) == 0:
            return test.UNCLEAR("Not tested. No resources found.")

        testRouteAction = outputList[0].findAcceptableTestRoute()
        activation = Activation()
        activation.addAction(testRouteAction)
        activation.type = SCHEDULED_RELATIVE_ACTIVATION
        activation.activationTimestamp = "5:0"
        activation.fireActivation()
        activation.checkLock()

        activation.delete() # added this line

        return test.PASS()
```

Changed files:
nmostesting/suites/IS0801Test.py

diff --git a/nmostesting/suites/IS0801Test.py b/nmostesting/suites/IS0801Test.py
index b4efd12..92e8b42 100644
--- a/nmostesting/suites/IS0801Test.py
+++ b/nmostesting/suites/IS0801Test.py
@@ -143,6 +143,7 @@ class IS0801Test(GenericTest):
         activation.activationTimestamp = "5:0"
         activation.fireActivation()
         activation.checkLock()
+        activation.delete()
 
         return test.PASS()
Repository: amwa-tv/nmos-testing
Issue: IS-04-01 test_05 may incorrectly report successful heartbeats

When performing the bulk of the IS-04-01 tests, various mock registries are enabled and interactions with them recorded. In one case I have observed, when the primary registry is enabled a Node first issued a heartbeat to it and received a 404 code. The Node then registered all of its resources before issuing its next heartbeat. Following this, a further heartbeat was issued, but at the wrong frequency. This should have been caught by `test_05`, but because two heartbeats had already been recorded, the third one happened after the primary registry had been disabled.

The code expects that heartbeats will only be recorded by the mock registry if a Node was already registered, but currently they are recorded whether a Node was registered or not.

This is definitely an edge case, but the failure to catch it makes it much harder to debug the registry failover tests like `test_15` and `test_16`.

Changed files:
nmostesting/mocks/Registry.py

diff --git a/nmostesting/mocks/Registry.py b/nmostesting/mocks/Registry.py
index 269ad6e..7cc32c1 100644
--- a/nmostesting/mocks/Registry.py
+++ b/nmostesting/mocks/Registry.py
@@ -175,9 +175,9 @@ def heartbeat(version, node_id):
     registry = REGISTRIES[flask.current_app.config["REGISTRY_INSTANCE"]]
     if not registry.enabled:
         abort(500)
-    # store raw request payload, in order to check for empty request bodies later
-    registry.heartbeat(request.headers, request.data, version, node_id)
     if node_id in registry.get_resources()["node"]:
+        # store raw request payload, in order to check for empty request bodies later
+        registry.heartbeat(request.headers, request.data, version, node_id)
         return jsonify({"health": int(time.time())})
     else:
         abort(404)
Repository: amwa-tv/nmos-testing
Issue: IS-08-01: test_14 only sends one action instead of two actions

The reordering constraint is supposed to send 2 channel changes that will eventually get blocked. 
```
blockOneAction = Action(
                    inputInstance.id,
                    targetOutputList[inputInstance.id].id,
                    inputChannelIndex,
                    outputChannelIndex
                )
blockTwoAction = Action(
                    inputInstance.id,
                    targetOutputList[inputInstance.id].id,
                    outputChannelIndex,
                    inputChannelIndex
                )
activation.addAction(blockOneAction)
activation.addAction(blockTwoAction)
```
However the test-suite only sends latter channel change due to the dictionary `.update` in `_actionObject(self)` in `activation.py` which overrides the first `blockOneAction`.

```
def _actionObject(self):
        actionObject = {}
        for action in self.actionList:
            actionObject.update(action.toJSON())
        return actionObject
```


Changed files:
nmostesting/suites/is08/action.py, nmostesting/suites/is08/activation.py

diff --git a/nmostesting/suites/is08/action.py b/nmostesting/suites/is08/action.py
index 7403be3..86a3d28 100644
--- a/nmostesting/suites/is08/action.py
+++ b/nmostesting/suites/is08/action.py
@@ -21,12 +21,10 @@ class Action():
         self.inputChannel = inputChannel
         self.outputChannel = outputChannel
 
-    def toJSON(self):
+    def channelJSON(self):
         return {
-            self.outputID: {
-                self.outputChannel: {
-                    "input": self.inputID,
-                    "channel_index": self.inputChannel
-                }
+            self.outputChannel: {
+                "input": self.inputID,
+                "channel_index": self.inputChannel
             }
         }
diff --git a/nmostesting/suites/is08/activation.py b/nmostesting/suites/is08/activation.py
index 492ebeb..5ddce25 100644
--- a/nmostesting/suites/is08/activation.py
+++ b/nmostesting/suites/is08/activation.py
@@ -44,7 +44,10 @@ class Activation:
     def _actionObject(self):
         actionObject = {}
         for action in self.actionList:
-            actionObject.update(action.toJSON())
+            if action.outputID not in actionObject:
+                actionObject[action.outputID] = action.channelJSON()
+            else:
+                actionObject[action.outputID].update(action.channelJSON())
         return actionObject
 
     def _buildPOSTObject(self):
Repository: amwa-tv/nmos-testing
Issue: IS05Utils.check_perform_immediate_activation fails at "if stagedParam == activeParam" comparision... 

...when "stagedParam" equals to "auto" 

In my case  websocket IS-07 sender is checked by test suite "IS-05-05, test_25" and fails

Here are the senders  constraints where the "auto" parameter ist taken from for the test

```json
[
    {
        "connection_authorization": {
            "enum": [
                false
            ]
        },
        "connection_uri": {
            "enum": [
                "auto",
                "ws://192.168.178.30:4491/x-nmos/events/v1.0/device_id/2840863c-f15d-43a7-969b-59b65f3716f6"
            ]
        },
        "ext_is_07_rest_api_url": {
            "enum": [
                "auto",
                "http://192.168.178.30:4490/x-nmos/events/v1.0/sources/e1126855-658f-47b7-b380-b2576faf347c/"
            ]
        },
        "ext_is_07_source_id": {
            "enum": [
                "auto",
                "e1126855-658f-47b7-b380-b2576faf347c"
            ]
        }
    }
]
```

this behavior  affects also tests 16, 27, 29

Changed files:
nmostesting/IS05Utils.py

diff --git a/nmostesting/IS05Utils.py b/nmostesting/IS05Utils.py
index 19af65b..49ecb3b 100644
--- a/nmostesting/IS05Utils.py
+++ b/nmostesting/IS05Utils.py
@@ -183,7 +183,7 @@ class IS05Utils(NMOSUtils):
                             return False, "Expected a dict to be returned from {} on leg {}, got a {}: {}".format(
                                 stagedUrl, i, type(response3), stagedParams)
                         msg = "Transport parameters did not transition to active during an imeddiate activation"
-                        if stagedParam == activeParam:
+                        if (stagedParam == activeParam if stagedParam != "auto" else "auto" != activeParam):
                             pass
                         else:
                             return False, msg
@@ -258,7 +258,7 @@ class IS05Utils(NMOSUtils):
                         except TypeError:
                             return False, "Expected a dict to be returned from {} on leg {}, " \\
                                           "got a {}: {}".format(stagedUrl, i, type(activeParams), stagedParams)
-                        if stagedParam == activeParam:
+                        if (stagedParam == activeParam if stagedParam != "auto" else "auto" != activeParam):
                             finished = True
                         else:
                             retries = retries + 1
@@ -351,7 +351,7 @@ class IS05Utils(NMOSUtils):
                         except TypeError:
                             return False, "Expected a dict to be returned from {} on leg {}, got a {}: " \\
                                           "{}".format(stagedUrl, i, type(activeParams), stagedParams)
-                        if activeParam == stagedParam:
+                        if (stagedParam == activeParam if stagedParam != "auto" else "auto" != activeParam):
                             finished = True
                         else:
                             retries = retries + 1
Repository: amwa-tv/nmos-testing
Issue: IS-04-01 test_13 should only test RTP receivers

I'm seeing `IS0401Test.test_13` fail on a WebSocket receiver. The test doesn't check the `transport` beforehand (only `format`) and doesn't check the returned status code before testing the response body.

(Follow-up to PR #283 which adjusts IS-05-01 tests for the new transports.)

Changed files:
IS0401Test.py

diff --git a/IS0401Test.py b/IS0401Test.py
index 30931a6..1760cba 100644
--- a/IS0401Test.py
+++ b/IS0401Test.py
@@ -658,6 +658,9 @@ class IS0401Test(GenericTest):
         try:
             formats_tested = []
             for receiver in receivers.json():
+                if not receiver["transport"].startswith("urn:x-nmos:transport:rtp"):
+                    continue
+
                 try:
                     stream_type = receiver["format"].split(":")[-1]
                 except TypeError:
@@ -697,7 +700,7 @@ class IS0401Test(GenericTest):
         except json.JSONDecodeError:
             return test.FAIL("Non-JSON response returned from Node API")
 
-        return test.UNCLEAR("Node API does not expose any Receivers")
+        return test.UNCLEAR("Node API does not expose any RTP Receivers")
 
     def test_14(self, test):
         """PUTing to a Receiver target resource with an empty JSON object payload is accepted and
@@ -708,15 +711,21 @@ class IS0401Test(GenericTest):
             return test.FAIL("Unexpected response from the Node API: {}".format(receivers))
 
         try:
-            if len(receivers.json()) > 0:
-                receiver = receivers.json()[0]
-                self.do_receiver_put(test, receiver["id"], {})
+            test_receiver = None
+            for receiver in receivers.json():
+                if not receiver["transport"].startswith("urn:x-nmos:transport:rtp"):
+                    continue
+                test_receiver = receiver
+                break
+
+            if test_receiver is not None:
+                self.do_receiver_put(test, test_receiver["id"], {})
 
                 time.sleep(API_PROCESSING_TIMEOUT)
 
-                valid, response = self.do_request("GET", self.node_url + "receivers/" + receiver["id"])
+                valid, response = self.do_request("GET", self.node_url + "receivers/" + test_receiver["id"])
                 if not valid:
-                    return test.FAIL("Unexpected response from the Node API: {}".format(receiver))
+                    return test.FAIL("Unexpected response from the Node API: {}".format(test_receiver))
 
                 receiver = response.json()
                 if receiver["subscription"]["sender_id"] is not None:
@@ -733,7 +742,7 @@ class IS0401Test(GenericTest):
         except json.JSONDecodeError:
             return test.FAIL("Non-JSON response returned from Node API")
 
-        return test.UNCLEAR("Node API does not expose any Receivers")
+        return test.UNCLEAR("Node API does not expose any RTP Receivers")
 
     def test_15(self, test):
         """Node correctly selects a Registration API based on advertised priorities"""
Repository: amwa-tv/nmos-testing
Issue: IS-04-01 test_20 shouldn't fail api.endpoints when testing Node API v1.0

Testing a Node API v1.0, reports "None of the Node 'api.endpoints' match the current protocol, IP/hostname and port".

Presumably the `if not found_api_endpoint` and `if not found_href` checks need to be protected by an `if self.is04_utils.compare_api_version(api["version"], "v1.0") >= 0:` check.


Changed files:
IS0401Test.py

diff --git a/IS0401Test.py b/IS0401Test.py
index 5b3251d..429899b 100644
--- a/IS0401Test.py
+++ b/IS0401Test.py
@@ -1038,13 +1038,13 @@ class IS0401Test(GenericTest):
         except json.JSONDecodeError:
             return test.FAIL("Non-JSON response returned from Node API")
 
-        if not found_api_endpoint:
-            return test.FAIL("None of the Node 'api.endpoints' match the current protocol, IP/hostname and port")
+        if self.is04_utils.compare_api_version(api["version"], "v1.1") >= 0:
+            if not found_api_endpoint:
+                return test.FAIL("None of the Node 'api.endpoints' match the current protocol, IP/hostname and port")
 
-        if not found_href:
-            return test.FAIL("None of the Node 'api.endpoints' match the Node 'href'")
+            if not found_href:
+                return test.FAIL("None of the Node 'api.endpoints' match the Node 'href'")
 
-        if self.is04_utils.compare_api_version(api["version"], "v1.1") >= 0:
             valid, response = self.do_request("GET", self.node_url + "devices")
             if not valid:
                 return test.FAIL("Unexpected response from the Node API: {}".format(response))
Repository: amwa-tv/nmos-testing
Issue: IS-04-03 test_01_node_mdns_with_txt fails 'api_auth' check when it shouldn't

Looks like zeroconf is automagically mapping the TXT record string value `false` to the Python `False`. We need to actually check the string value, since the spec is fierce about lowercase (and who knows what else zeroconf may evaluate and turn into a Boolean).

Changed files:
IS0402Test.py, IS0401Test.py, IS0403Test.py

diff --git a/IS0401Test.py b/IS0401Test.py
index 5b3251d..dd287cd 100644
--- a/IS0401Test.py
+++ b/IS0401Test.py
@@ -605,7 +605,8 @@ class IS0401Test(GenericTest):
                 if self.is04_utils.compare_api_version(api["version"], "v1.3") >= 0:
                     if "api_auth" not in properties:
                         return test.FAIL("No 'api_auth' TXT record found in Node API advertisement.")
-                    elif properties["api_auth"] not in ["true", "false"]:
+                    elif not isinstance(properties["api_auth"], bool):
+                        # zeroconf translates 'true' to True and 'false' to False automatically
                         return test.FAIL("API authorization ('api_auth') TXT record is not one of 'true' or 'false'.")
 
                 return test.PASS()
diff --git a/IS0402Test.py b/IS0402Test.py
index 4cd6f96..78a300a 100644
--- a/IS0402Test.py
+++ b/IS0402Test.py
@@ -106,7 +106,8 @@ class IS0402Test(GenericTest):
                 if self.is04_reg_utils.compare_api_version(api["version"], "v1.3") >= 0:
                     if "api_auth" not in properties:
                         return test.FAIL("No 'api_auth' TXT record found in {} advertisement.".format(api["name"]))
-                    elif properties["api_auth"] not in ["true", "false"]:
+                    elif not isinstance(properties["api_auth"], bool):
+                        # zeroconf translates 'true' to True and 'false' to False automatically
                         return test.FAIL("API authorization ('api_auth') TXT record is not one of 'true' or 'false'.")
 
                 return test.PASS()
diff --git a/IS0403Test.py b/IS0403Test.py
index e7c55ed..cdf4d02 100644
--- a/IS0403Test.py
+++ b/IS0403Test.py
@@ -83,7 +83,8 @@ class IS0403Test(GenericTest):
                 if self.is04_utils.compare_api_version(api["version"], "v1.3") >= 0:
                     if "api_auth" not in properties:
                         return test.FAIL("No 'api_auth' TXT record found in Node API advertisement.")
-                    elif properties["api_auth"] not in ["true", "false"]:
+                    elif not isinstance(properties["api_auth"], bool):
+                        # zeroconf translates 'true' to True and 'false' to False automatically
                         return test.FAIL("API authorization ('api_auth') TXT record is not one of 'true' or 'false'.")
 
                 return test.PASS()
Repository: amwa-tv/nmos-testing
Issue: Redundant sender can have more than two legs

Accoding to ST 2022-7:2019 a redundant sender can have at least two transmitting legs. 

https://github.com/AMWA-TV/nmos-testing/blob/228f2a21fca58351f4f68544d46f1ec3482eba9d/IS05Utils.py#L43

This constraints test 38 of IS-05-01 to two legs.

Changed files:
IS05Utils.py

diff --git a/IS05Utils.py b/IS05Utils.py
index 08cef3c..b1383bb 100644
--- a/IS05Utils.py
+++ b/IS05Utils.py
@@ -40,7 +40,6 @@ class IS05Utils(NMOSUtils):
 
     def check_num_legs(self, url, type, uuid):
         """Checks the number of legs present on a given sender/receiver"""
-        max = 2
         min = 1
         constraintsUrl = url + "constraints/"
         stagedUrl = url + "staged/"
@@ -69,10 +68,7 @@ class IS05Utils(NMOSUtils):
                     except KeyError:
                         return False, "Could not find transport params in object from {}, got {}".format(activeUrl,
                                                                                                          active)
-                    if len(constraints) <= max:
-                        pass
-                    else:
-                        return False, "{} {} has too many legs".format(type, uuid)
+
                     if len(constraints) >= min:
                         pass
                     else: