Repository: StackStorm/st2

Issue:
Config.schema's are not applied if there are no configs in /opt/stackstorm/configs/
Hi,

This a odd one, just built a new 2.2 dev instance to finish up stackstorm-networking_utils#1 and I've found that the following schema for `networking_utils`:

```yaml
isp_db:
  description: "Path to ISP GeoIP2 database"
  type: "string"
  required: true
  default: "/opt/geoip2/GeoIP2-ISP.mmdb"
city_db:
  description: "Path to City GeoIP2 database"
  type: "string"
  required: true
  default: "/opt/geoip2/GeoLite2-City.mmdb"
```

Is only applied if at least 1 config file exists in `/opt/stackstorm/configs/` (for any pack). Without this I was key errors on `isp_db` in my `geoip` action.


Discussion:
@jjm As I understand it, schema only applies to configs in `/opt/stackstorm/configs/`. If you have a config in `/opt/stackstorm/packs/<my_pack>/config.yaml`, then the schema doesn't apply.

How was your system setup? With just `/opt/stackstorm/packs/networking_utils/config.yaml`? Where were you seeing the key errors?
Yeah, what @LindsayHill said - schema is only applied to new style configs located at `/opt/stackstorm/configs/<pack name>.yaml`.

Or is there something else going on / context we are missing?
Yes, what @LindsayHill said is what I believe to be the case too @Kami. This is issue a little more complex, apologies I must have rushed the write up last night.

The following steps should reproduce the issue:

- Fresh install of ST2, with no files in `/opt/stackstorm/configs/`.
- Have a pack that only has `config.schema.yaml` and not a `config.yaml`.
- Load configs via `st2-register-content --register-configs` and see `0` config's are loaded.
- Run a action (e.g. `networking_utils.geoip`) that has a config schema with required properties.
- Watch the action raise exceptions about lack of keys.
- Create a config file in `/opt/stackstorm/configs/` for another pack (e.g. `orion`).
- Load configs via `st2-register-content --register-configs` and see `1` config's are loaded.
- Rerun the action and watch it get the properties that are required and have a default.

In short, should all `config.schema.yaml` should be loaded if one exists and the pack does not contain it's own `config.yaml`?. As it currently stands it seems that `config.schema.yaml` files are _only_ loaded if at least 1 config file exists in  `/opt/stackstorm/configs/` .

It may be clearer if `st2-register-content --register-configs` reported the number of schema's loaded along with the number of config file and if any schema's were not loaded due to the existence of `config.yaml` in the pack.

Hope this makes it clearer.

@jjm Thanks for a more detailed write-up.

The behavior you described is a normal and expected behavior.

First of all, config schemas and configs are two different objects / resources.

Config schemas are registered in the same way packs are registered - they are "implicitly" registered every time `st2-register-content` script is ran.

``--register-configs`` flag causes it to register config files itself so we don't print number of config schemas and packs which are registered (they are registered implicitly on each register content script run).

As packs and config schemas are registered implicitly on each run, we don't print "x packs" and "x config schemas" registered because it would cause too much clutter (you will see which packs and schemas are registered though if you run register-content script with `--debug` flag or set debug log level).

So in short, yes, config schemas are registered even if you don't have any configs since they are a special case like pack (we only provide read-only APIs for PackDB and ConfigSchemaDB objects and those objects can't be manipulated via the API like other pack resources).

If you think there are ways we could make this more clear and obvious, I'm open to suggestions (more / better docs, perhaps changing messaging of the register content script).

Resolving patch:
diff --git a/st2common/st2common/util/config_loader.py b/st2common/st2common/util/config_loader.py
--- a/st2common/st2common/util/config_loader.py
+++ b/st2common/st2common/util/config_loader.py
@@ -20,6 +20,7 @@
 from oslo_config import cfg
 
 from st2common import log as logging
+from st2common.models.db.pack import ConfigDB
 from st2common.persistence.pack import ConfigSchema
 from st2common.persistence.pack import Config
 from st2common.content import utils as content_utils
@@ -72,8 +73,10 @@ def get_config(self):
         try:
             config_db = Config.get_by_pack(value=self.pack_name)
         except StackStormDBObjectNotFoundError:
-            # Corresponding pack config doesn't exist, return early
-            return result
+            # Corresponding pack config doesn't exist. We set config_db to an empty config so
+            # that the default values from config schema are still correctly applied even if
+            # pack doesn't contain a config.
+            config_db = ConfigDB(pack=self.pack_name, values={})
 
         try:
             config_schema_db = ConfigSchema.get_by_pack(value=self.pack_name)


Test patch:
diff --git a/st2common/tests/unit/test_config_loader.py b/st2common/tests/unit/test_config_loader.py
--- a/st2common/tests/unit/test_config_loader.py
+++ b/st2common/tests/unit/test_config_loader.py
@@ -15,17 +15,18 @@
 
 from st2common.persistence.pack import Config
 from st2common.models.db.pack import ConfigDB
+from st2common.exceptions.db import StackStormDBObjectNotFoundError
 from st2common.services.config import set_datastore_value_for_config_key
 from st2common.util.config_loader import ContentPackConfigLoader
 
-from st2tests.base import DbTestCase
+from st2tests.base import CleanDbTestCase
 
 __all__ = [
     'ContentPackConfigLoaderTestCase'
 ]
 
 
-class ContentPackConfigLoaderTestCase(DbTestCase):
+class ContentPackConfigLoaderTestCase(CleanDbTestCase):
     register_packs = True
     register_pack_configs = True
 
@@ -92,6 +93,21 @@ def test_get_config_default_value_from_config_schema_is_used(self):
         config = loader.get_config()
         self.assertEqual(config['region'], 'us-west-1')
 
+    def test_default_values_from_schema_are_used_when_no_config_exists(self):
+        pack_name = 'dummy_pack_5'
+        config_db = Config.get_by_pack(pack_name)
+
+        # Delete the existing config loaded in setUp
+        config_db = Config.get_by_pack(pack_name)
+        config_db.delete()
+
+        # Verify config has been deleted from the database
+        self.assertRaises(StackStormDBObjectNotFoundError, Config.get_by_pack, pack_name)
+
+        loader = ContentPackConfigLoader(pack_name=pack_name)
+        config = loader.get_config()
+        self.assertEqual(config['region'], 'default-region-value')
+
     def test_get_config_nested_schema_default_values_from_config_schema_are_used(self):
         # Special case for more complex config schemas with attributes ntesting.
         # Validate that the default values are also used for one level nested object properties.