Repository: opsdroid/opsdroid

Issue:
Add config validation
Currently we are not doing a very good job of validating config before loading opsdroid.

An example of this is that we have [very old and now invalid configs](https://github.com/opsdroid/opsdroid/blob/bc95a2bd4d7cf921974f4ee01ec1b51003249bc3/tests/configs/minimal.yaml) in our tests and they are still passing. In that config `skills` is a dictionary, which was changed to a list a long time ago.

We should check our configs are valid before starting to load.


Discussion:
Hello, I'd love to help out on this project. Do you think this is a good place to start and if so can you say more about what need to be done for the config tests?
Hello @bicep welcome to the project. Can't say exactly if this is the right issue for you - I guess it depends on how well you know yaml and pyyaml 😄 

Basically we are loading the `config.yaml` but there is little validation done on this file (for examples, adding three spaces instead of two will make the config break).


As for the tests, what needs to change is from a dictionary into a list and update them - also checking if things will break or not. 

I hope this will helps a bit. If you think this is a bit out of scope of what you can do you can check the issues with [good first issue](https://github.com/opsdroid/opsdroid/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) or the [intermediate level](https://github.com/opsdroid/opsdroid/issues?q=is%3Aissue+is%3Aopen+label%3Aintermediate).

Let me know if you need any help with anything 😄 👍 
Hi! I'd like to try fix this issue. Could you tell me a little bit more about this? When we need to check config.yaml ? Any code snippet or hint?
I've found some class named Loader, I think it should be there.
We could take a leaf out of Home Assistants book here and have connectors and skills define a schema for their config. 

See https://developers.home-assistant.io/docs/en/development_validation.html and https://pypi.org/project/voluptuous/ is the lib they use.
Hello, I would like to handle this issue. Is it still actual?
Heya and welcome!

Yeah this issue is still actual and would be great if you could tackle it. We have just updated pyyaml to the latest version 5.1.1(2?) but home assistant is still using 3-4.

Please let me know if you need any help with this issue.

Thank you 😄👍

Resolving patch:
diff --git a/opsdroid/loader.py b/opsdroid/loader.py
--- a/opsdroid/loader.py
+++ b/opsdroid/loader.py
@@ -2,6 +2,7 @@
 
 # pylint: disable=too-many-branches
 
+import yamale
 import importlib
 import importlib.util
 import json
@@ -351,10 +352,20 @@ def include_constructor(loader, node):
         try:
             with open(config_path, "r") as stream:
                 _LOGGER.info(_("Loaded config from %s."), config_path)
+                schema_path = os.path.abspath("opsdroid/configuration/schema.yaml")
+                schema = yamale.make_schema(schema_path)
+                data = yamale.make_data(config_path)
+                yamale.validate(schema, data)
                 return yaml.safe_load(stream)
+
+        except ValueError as error:
+            _LOGGER.critical(error)
+            sys.exit(1)
+
         except yaml.YAMLError as error:
             _LOGGER.critical(error)
             sys.exit(1)
+
         except FileNotFoundError as error:
             _LOGGER.critical(error)
             sys.exit(1)


Test patch:
diff --git a/tests/configs/broken_without_connectors.yaml b/tests/configs/broken_without_connectors.yaml
new file mode 100644
--- /dev/null
+++ b/tests/configs/broken_without_connectors.yaml
@@ -0,0 +1,3 @@
+skills:
+  - name: hello
+  - name: seen
diff --git a/tests/configs/full_broken.yaml b/tests/configs/full_broken.yaml
new file mode 100644
--- /dev/null
+++ b/tests/configs/full_broken.yaml
@@ -0,0 +1,35 @@
+connectors:
+
+  - name: # WRONG CONFIG: the name field is missing
+    bot-name: "mybot"
+    max-connections: 10
+    connection-timeout: 10
+
+  - name: facebook
+    verify-token: aabbccddee
+    page-access-token: aabbccddee112233445566
+    bot-name: "mybot"
+
+  - name: github
+    github-token: aaabbbcccdddeee111222333444
+
+databases:
+  - name: mongo
+    host: "my host"
+    port:  "12345"
+    database: "mydatabase"
+
+  - name: redis
+    host: "my host"
+    port:  "12345"
+    database: "7"
+
+  - name: sqlite
+    file: "my_file.db"
+    table: "my_table"
+
+skills:
+  - name: dance
+  - name: hello
+  - name: loudnoises
+  - name: seen
diff --git a/tests/configs/full_valid.yaml b/tests/configs/full_valid.yaml
new file mode 100644
--- /dev/null
+++ b/tests/configs/full_valid.yaml
@@ -0,0 +1,31 @@
+welcome-message: true
+
+connectors:
+
+  - name: websocket
+    bot-name: "mybot"
+    max-connections: 10
+    connection-timeout: 10
+
+databases:
+
+  - name: mongo
+    host: "my host"
+    port:  "12345"
+    database:  "mydatabase"
+
+parsers:
+
+  - name: dialogflow
+    access-token: "exampleaccesstoken123"
+    min-score: 0.6
+
+skills:
+
+  - name: dance
+
+  - name: hello
+
+  - name: loudnoises
+
+  - name: seen
diff --git a/tests/configs/minimal.yaml b/tests/configs/minimal.yaml
--- a/tests/configs/minimal.yaml
+++ b/tests/configs/minimal.yaml
@@ -1,6 +1,6 @@
 connectors:
-  shell:
+  - name: shell
 
 skills:
-  hello:
-  seen:
+  - name: hello
+  - name: seen
diff --git a/tests/configs/minimal_2.yaml b/tests/configs/minimal_2.yaml
--- a/tests/configs/minimal_2.yaml
+++ b/tests/configs/minimal_2.yaml
@@ -1,9 +1,9 @@
 connectors:
-  shell:
+  - name: shell
 
 skills:
-  hello:
-  seen:
+  - name: hello
+  - name: seen
 
 databases:
 
diff --git a/tests/configs/minimal_with_envs.yaml b/tests/configs/minimal_with_envs.yaml
--- a/tests/configs/minimal_with_envs.yaml
+++ b/tests/configs/minimal_with_envs.yaml
@@ -1 +1,6 @@
-test: $ENVVAR
+
+connectors:
+  - name: $ENVVAR
+
+skills:
+  - name: hello
diff --git a/tests/configs/minimal_with_include.yaml b/tests/configs/minimal_with_include.yaml
--- a/tests/configs/minimal_with_include.yaml
+++ b/tests/configs/minimal_with_include.yaml
@@ -1 +1,2 @@
 !include minimal.yaml
+
diff --git a/tests/configs/valid_case_sensitivity.yaml b/tests/configs/valid_case_sensitivity.yaml
new file mode 100644
--- /dev/null
+++ b/tests/configs/valid_case_sensitivity.yaml
@@ -0,0 +1,31 @@
+welcome-message: true
+
+connectors:
+
+  - name: Websocket
+    bot-name: "mybot"
+    max-connections: 10
+    connection-timeout: 10
+
+databases:
+
+  - name: Mongo
+    host: "my host"
+    port:  "12345"
+    database:  "mydatabase"
+
+parsers:
+
+  - name: Dialogflow
+    access-token: "exampleaccesstoken123"
+    min-score: 0.6
+
+skills:
+
+  - name: dance
+
+  - name: hello
+
+  - name: loudnoises
+
+  - name: seen
diff --git a/tests/configs/valid_without_db_and_parsers.yaml b/tests/configs/valid_without_db_and_parsers.yaml
new file mode 100644
--- /dev/null
+++ b/tests/configs/valid_without_db_and_parsers.yaml
@@ -0,0 +1,20 @@
+# CORRECT CONFIG: the databases and parsers are optional
+
+welcome-message: true
+
+connectors:
+
+  - name: websocket
+    bot-name: "mybot"
+    max-connections: 10
+    connection-timeout: 10
+
+skills:
+
+  - name: dance
+
+  - name: hello
+
+  - name: loudnoises
+
+  - name: seen
diff --git a/tests/configs/valid_without_wellcome_message.yaml b/tests/configs/valid_without_wellcome_message.yaml
new file mode 100644
--- /dev/null
+++ b/tests/configs/valid_without_wellcome_message.yaml
@@ -0,0 +1,32 @@
+# CORRECT CONFIG: the file is being checked despite the absence of a welcome message
+
+connectors:
+
+  - name: websocket
+    bot-name: "mybot"
+    max-connections: 10
+    connection-timeout: 10
+
+databases:
+
+  - name: mongo
+    host: "my host"
+    port:  "12345"
+    database:  "mydatabase"
+
+parsers:
+
+  - name: dialogflow
+    access-token: "exampleaccesstoken123"
+    min-score: 0.6
+
+
+skills:
+
+  - name: dance
+
+  - name: hello
+
+  - name: loudnoises
+
+  - name: seen
diff --git a/tests/test_loader.py b/tests/test_loader.py
--- a/tests/test_loader.py
+++ b/tests/test_loader.py
@@ -41,6 +41,51 @@ def test_load_config_file(self):
         )
         self.assertIsNotNone(config)
 
+    def test_load_config_valid(self):
+        opsdroid, loader = self.setup()
+        config = loader.load_config_file(
+            [os.path.abspath("tests/configs/full_valid.yaml")]
+        )
+        self.assertIsNotNone(config)
+
+    def test_load_config_valid_without_wellcome_message(self):
+        opsdroid, loader = self.setup()
+        config = loader.load_config_file(
+            [os.path.abspath("tests/configs/valid_without_wellcome_message.yaml")]
+        )
+        self.assertIsNotNone(config)
+
+    def test_load_config_valid_without_db_and_parsers(self):
+        opsdroid, loader = self.setup()
+        config = loader.load_config_file(
+            [os.path.abspath("tests/configs/valid_without_db_and_parsers.yaml")]
+        )
+        self.assertIsNotNone(config)
+
+    def test_load_config_broken_without_connectors(self):
+        opsdroid, loader = self.setup()
+        with self.assertRaises(SystemExit) as cm:
+            config = loader.load_config_file(
+                [os.path.abspath("tests/configs/broken_without_connectors.yaml")]
+            )
+        self.assertEqual(cm.exception.code, 1)
+
+    def test_load_config_valid_case_sensitivity(self):
+        opsdroid, loader = self.setup()
+        config = loader.load_config_file(
+            [os.path.abspath("tests/configs/valid_case_sensitivity.yaml")]
+        )
+        self.assertIsNotNone(config)
+
+    def test_load_config_broken(self):
+        opsdroid, loader = self.setup()
+
+        with self.assertRaises(SystemExit) as cm:
+            config = loader.load_config_file(
+                [os.path.abspath("tests/configs/full_broken.yaml")]
+            )
+        self.assertEqual(cm.exception.code, 1)
+
     def test_load_config_file_2(self):
         opsdroid, loader = self.setup()
         config = loader.load_config_file(
@@ -67,6 +112,7 @@ def test_load_exploit(self):
             self.assertRaises(YAMLError)
             unittest.main(exit=False)
 
+    @unittest.skip("old config type fails validation #770")
     def test_load_config_file_with_include(self):
         opsdroid, loader = self.setup()
         config = loader.load_config_file(
@@ -87,13 +133,14 @@ def test_yaml_load_exploit(self):
             # If the command in exploit.yaml is echoed it will return 0
             self.assertNotEqual(config, 0)
 
+    @unittest.skip("old config type fails validation #770")
     def test_load_config_file_with_env_vars(self):
         opsdroid, loader = self.setup()
-        os.environ["ENVVAR"] = "test"
+        os.environ["ENVVAR"] = "shell"
         config = loader.load_config_file(
             [os.path.abspath("tests/configs/minimal_with_envs.yaml")]
         )
-        self.assertEqual(config["test"], os.environ["ENVVAR"])
+        self.assertEqual(config["connectors"][0]["name"], os.environ["ENVVAR"])
 
     def test_create_default_config(self):
         test_config_path = os.path.join(