Repository: StackStorm/st2

Issue:
RBAC on unstable fails to load
## SUMMARY

Provide a quick summary of your bug report.

### STACKSTORM VERSION

Paste the output of ``st2 --version``:

##### OS, environment, install method

Ansible CI/CD has been failing for 3 days on the smoke tests at the RBAC apply defintions

## Steps to reproduce the problem

Perform a ansible install with environment -e st2_rbac_enable=True. At the point it loads the rbac definition file with the smoke test resource it fails.
Think this is probably true for any RBAC on unstable.

## Expected Results

Load should complete.

## Actual Results

The ansible script fails on the rbac. Reproducing this on a clean VM and running the ansible install then we see error:
```
2022-07-22 11:29:13,398 INFO [-] Loading role definitions from "/opt/stackstorm/rbac/roles/"
Traceback (most recent call last):
  File "/bin/st2-apply-rbac-definitions", line 15, in <module>
    sys.exit(apply_rbac_definitions.main(sys.argv[1:]))
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2rbac_backend/cmd/apply_rbac_definitions.py", line 60, in main
    apply_definitions()
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2rbac_backend/cmd/apply_rbac_definitions.py", line 42, in apply_definitions
    result = loader.load()
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2rbac_backend/loader.py", line 61, in load
    result["roles"] = self.load_role_definitions()
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2rbac_backend/loader.py", line 79, in load_role_definitions
    role_definition_api = self.load_role_definition_from_file(file_path=file_path)
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2rbac_backend/loader.py", line 159, in load_role_definition_from_file
    role_definition_api = role_definition_api.validate()
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/models/api/rbac.py", line 170, in validate
    cleaned = super(RoleDefinitionFileFormatAPI, self).validate()
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/models/api/base.py", line 75, in validate
    cleaned = util_schema.validate(
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/util/schema/__init__.py", line 370, in validate
    instance = assign_default_values(instance=instance, schema=schema)
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/util/schema/__init__.py", line 255, in assign_default_values
    instance[property_name] = assign_default_values(
  File "/opt/stackstorm/st2/lib/python3.8/site-packages/st2common/util/schema/__init__.py", line 249, in assign_default_values
    array_instance = instance.get(property_name, None)
AttributeError: 'list' object has no attribute 'get'
```

There is only one role defined, and it looks good:
```
$ cat /opt/stackstorm/rbac/roles/smoke_tests_rbac_basic.yaml 
---

name: smoke_tests_rbac_basic
description: This role has access only to action core.local in pack 'core'
permission_grants:
  - permission_types:
    - action_execute
    - action_view
    resource_uid: action:core:local
  - permission_types:
    - runner_type_list

```


Discussion:
@cognifloyd  Could this be related to the output schema changes that went in via: https://github.com/StackStorm/st2/pull/5319

It's around the right time, as these tests were reliable before. And it looks like similar area of code. Wondering if the RBAC schema is incorrect as per the updates to the schema checking?

Do we need to define the schemas that are defined in code, e.g. https://github.com/StackStorm/st2/blob/87260921344fd4ca4a7d5c692595fc920eea2b16/st2common/st2common/models/api/rbac.py#L116

Or the problem migth be that the schema has a default of [] on the permission_types, and the code for setting defaults doesn't cope with setting that?
@amanda11 wow. So, it looks like this bug was present before, BUT by fixing a different bug, I uncovered this bug.

Looks like:
- schema for an array of objects
    - a property in the items schema is also an array
        - :boom:

Before, it was only applying defaults to each of the items schema properties if, and only if, the property's schema included `properties` which effectively meant it only set defaults for objects and ignored setting defaults within array properties before.

I think I see what the fix requires, hopefully I can submit a PR to add tests for that scenario and fix that today or tomorrow.
OK. I think the best way to fix this is to merge these two assign_default_values implementations:
- https://github.com/StackStorm/st2/blob/3e1a052f27d8deb2d53402ee60e75126e10f6c69/st2common/st2common/util/schema/__init__.py#L202-L205
- https://github.com/StackStorm/st2/blob/7110c75cd362c1fcc3ba9dd8174ecb655d54b662/st2common/st2common/util/config_loader.py#L297-L305

I've spent a lot of time revising the pack config loader and ensuring it can handle many different schema types in #5225 and #5321, so I'd like to move that implementation from `st2common.util.config_loader.ContentPackConfigLoader` to `st2common.util.schema`. 

Resolving patch:
diff --git a/st2common/st2common/util/schema/__init__.py b/st2common/st2common/util/schema/__init__.py
--- a/st2common/st2common/util/schema/__init__.py
+++ b/st2common/st2common/util/schema/__init__.py
@@ -16,7 +16,6 @@
 from __future__ import absolute_import
 
 import os
-from typing import Mapping, Sequence
 
 import six
 import jsonschema
@@ -210,11 +209,28 @@ def assign_default_values(instance, schema):
     if not instance_is_dict and not instance_is_array:
         return instance
 
+    schema_type = schema.get("type", None)
+
+    if instance_is_dict and schema_type == "object":
+        return _assign_default_values_object(instance, schema)
+
+    elif instance_is_array and schema_type == "array":
+        return _assign_default_values_array(instance, schema)
+
+    return instance
+
+
+def _assign_default_values_object(instance, schema):
+    # assert is_attribute_type_object(schema.get("type"))
+    if not isinstance(instance, dict):
+        return instance
+
+    # TODO: handle defaults in patternProperties and additionalProperties
     properties = schema.get("properties", {})
     dependencies = schema.get("dependencies", {})
 
-    for property_name, property_data in six.iteritems(properties):
-        has_default_value = "default" in property_data
+    for property_name, property_schema in properties.items():
+        has_default_value = "default" in property_schema
         # only populate default if dependencies are met
         # eg: exclusiveMaximum depends on maximum which does not have a default.
         #     so we don't want to apply exclusiveMaximum's default unless maximum.
@@ -228,54 +244,95 @@ def assign_default_values(instance, schema):
                     # do not apply this default.
                     has_default_value = False
                     break
-        default_value = property_data.get("default", None)
+
+        default_value = property_schema.get("default", None)
 
         # Assign default value on the instance so the validation doesn't fail if requires is true
         # but the value is not provided
-        if has_default_value:
-            if instance_is_dict and instance.get(property_name, None) is None:
-                instance[property_name] = default_value
-            elif instance_is_array:
-                for index, _ in enumerate(instance):
-                    if instance[index].get(property_name, None) is None:
-                        instance[index][property_name] = default_value
+        if has_default_value and instance.get(property_name, None) is None:
+            instance[property_name] = default_value
+
+        attribute_instance = instance.get(property_name, None)
+        if attribute_instance is None:
+            # Note: We don't perform subschema assignment if no value is provided
+            continue
 
         # Support for nested properties (array and object)
-        attribute_type = property_data.get("type", None)
-        schema_items = property_data.get("items", {})
+        attribute_type = property_schema.get("type", None)
 
         # Array
-        if is_attribute_type_array(attribute_type) and schema_items:
-            array_instance = instance.get(property_name, None)
-            # Note: We don't perform subschema assignment if no value is provided
-            if array_instance is not None:
-                if isinstance(schema_items, Mapping) and schema_items.get(
-                    "properties", {}
-                ):
-                    instance[property_name] = assign_default_values(
-                        instance=array_instance, schema=schema_items
-                    )
-                elif array_instance and isinstance(schema_items, Sequence):
-                    array_instance_count = len(array_instance)
-                    for i, item_schema in enumerate(schema_items):
-                        if i > array_instance_count:
-                            break
-                        instance[property_name][i] = assign_default_values(
-                            instance=array_instance[i], schema=item_schema
-                        )
+        if is_attribute_type_array(attribute_type):
+            instance[property_name] = _assign_default_values_array(
+                instance=attribute_instance,
+                schema=property_schema,
+            )
 
         # Object
-        if is_attribute_type_object(attribute_type) and property_data.get(
-            "properties", {}
-        ):
-            object_instance = instance.get(property_name, None)
-            object_schema = schema["properties"][property_name]
+        elif is_attribute_type_object(attribute_type):
+            instance[property_name] = _assign_default_values_object(
+                instance=attribute_instance,
+                schema=property_schema,
+            )
 
-            if object_instance is not None:
-                # Note: We don't perform subschema assignment if no value is provided
-                instance[property_name] = assign_default_values(
-                    instance=object_instance, schema=object_schema
-                )
+    return instance
+
+
+def _assign_default_values_array(instance, schema):
+    # assert is_attribute_type_array(schema.get("type"))
+    if not isinstance(instance, list):
+        return instance
+
+    # make sure we don't mess up the original schema
+    schema = fast_deepcopy_dict(schema)
+
+    # TODO: handle defaults in additionalItems
+    items = schema.get("items", [])
+
+    if not items:
+        return instance
+
+    # items is either
+    # - a single schema for all items, or
+    # - a list of schemas (a positional schemaArray).
+    if isinstance(items, dict):
+        items = [items] * len(instance)
+
+    # assert isinstance(items, list)
+
+    for index, item_schema in enumerate(items):
+        has_instance_value = index < len(instance)
+        if not has_instance_value:
+            break
+
+        has_default_value = "default" in item_schema
+        default_value = item_schema.get("default", None)
+
+        # Assign default value in the instance so the validation doesn't fail if requires is true
+        # but the value is not provided
+        if has_default_value and instance[index] is None:
+            instance[index] = default_value
+
+        item_instance = instance[index]
+        if item_instance is None:
+            # Note: We don't perform subschema assignment if no value is provided
+            continue
+
+        # Support for nested properties (array and object)
+        item_type = item_schema.get("type", None)
+
+        # Array
+        if is_attribute_type_array(item_type):
+            instance[index] = _assign_default_values_array(
+                instance=item_instance,
+                schema=item_schema,
+            )
+
+        # Object
+        if is_attribute_type_object(item_type):
+            instance[index] = _assign_default_values_object(
+                instance=item_instance,
+                schema=item_schema,
+            )
 
     return instance
 
@@ -366,6 +423,7 @@ def validate(
     if use_default and allow_default_none:
         schema = modify_schema_allow_default_none(schema=schema)
 
+    # TODO: expand default handling to more than just objects
     if use_default and schema_type == "object" and instance_is_dict:
         instance = assign_default_values(instance=instance, schema=schema)
 


Test patch:
diff --git a/st2common/tests/unit/test_json_schema.py b/st2common/tests/unit/test_json_schema.py
--- a/st2common/tests/unit/test_json_schema.py
+++ b/st2common/tests/unit/test_json_schema.py
@@ -143,6 +143,34 @@
     },
 }
 
+TEST_SCHEMA_6 = {
+    "additionalProperties": False,
+    "title": "foo",
+    "description": "Nested array-object-array-string structure.",
+    "type": "object",
+    "properties": {
+        "arg_optional_type_array": {
+            "description": "like rbac role permission_grants",
+            "type": "array",
+            "items": {
+                "description": "like a permission_grant",
+                "type": "object",
+                "properties": {
+                    "item_arg_optional_type_array": {
+                        "description": "like rbac role grant permission_types",
+                        "type": "array",
+                        "items": {
+                            "type": "string",
+                            "enum": ["one", "two", "three"],
+                        },
+                        "default": [],
+                    },
+                },
+            },
+        },
+    },
+}
+
 
 class JSONSchemaTestCase(TestCase):
     def test_use_default_value(self):
@@ -412,3 +440,44 @@ def test_is_attribute_type_object(self):
         self.assertFalse(
             util_schema.is_attribute_type_object(array_type_property.get("type"))
         )
+
+    def test_nested_schemas(self):
+        validator = util_schema.get_validator()
+
+        # allow empty Array
+        instance = {"arg_optional_type_array": []}
+        util_schema.validate(
+            instance=instance, schema=TEST_SCHEMA_6, cls=validator, use_default=True
+        )
+
+        # allow Array with one item with None
+        instance = {"arg_optional_type_array": [{"item_arg_optional_type_array": None}]}
+        util_schema.validate(
+            instance=instance, schema=TEST_SCHEMA_6, cls=validator, use_default=True
+        )
+
+        # allow Array with one item with empty array
+        instance = {"arg_optional_type_array": [{"item_arg_optional_type_array": []}]}
+        util_schema.validate(
+            instance=instance, schema=TEST_SCHEMA_6, cls=validator, use_default=True
+        )
+
+        # allow Array with one item with Array with one string
+        instance = {
+            "arg_optional_type_array": [{"item_arg_optional_type_array": ["one"]}]
+        }
+        util_schema.validate(
+            instance=instance, schema=TEST_SCHEMA_6, cls=validator, use_default=True
+        )
+
+        # allow Array with multiple items
+        instance = {
+            "arg_optional_type_array": [
+                {"item_arg_optional_type_array": None},
+                {"item_arg_optional_type_array": ["one"]},
+                {"item_arg_optional_type_array": ["two", "three"]},
+            ]
+        }
+        util_schema.validate(
+            instance=instance, schema=TEST_SCHEMA_6, cls=validator, use_default=True
+        )