Repository: opsdroid/opsdroid

Issue:
Parsers raising KeyError if enabled flag is not set in config
<!-- Before you post an issue or if you are unsure about something join our matrix channel https://riot.im/app/#/room/#opsdroid-general:matrix.org and ask away! We are more than happy to help you. -->
# Description
While testing the new version of dialogflow, the parser breaks opsdroid if we don't have the `enabled` flag set on the config. Oddly enough when opsdroid loads dialogflow we can see that enabled is being set by `opsdroid.loader.setup_module_config` 

```python
INFO opsdroid.loader: {'name': 'dialogflow', 'module': '', 'project-id': 'test-ddd33', 'type': 'parsers', 'enabled': True, 'entrypoint': None, 'is_builtin': ModuleSpec(name='opsdroid.parsers.dialogflow', loader=<_frozen_importlib_external.SourceFileLoader object at 0x108bf8e10>, origin='/Users/fabiorosado/Documents/GitHub.tmp/opsdroid/opsdroid/parsers/dialogflow.py'), 'module_path': 'opsdroid.parsers.dialogflow', 'install_path': '/Users/fabiorosado/Library/Application Support/opsdroid/opsdroid-modules/parsers/dialogflow', 'branch': 'master'}
```

## Steps to Reproduce
- Setup dialogflow flow
- run opsdroid
- exception occurs


## Expected Functionality
- opsdroid should parse message


## Experienced Functionality
KeyError exception raised on `opsdroid.core.get_ranked_skills`.

<details>
<summary>Debug Log</summary>

```python
INFO opsdroid.logging: ========================================
INFO opsdroid.logging: Started opsdroid v0.8.1+833.g5c8dbb7.dirty.
INFO opsdroid: ========================================
INFO opsdroid: You can customise your opsdroid by modifying your configuration.yaml.
INFO opsdroid: Read more at: http://opsdroid.readthedocs.io/#configuration
INFO opsdroid: Watch the Get Started Videos at: http://bit.ly/2fnC0Fh
INFO opsdroid: Install Opsdroid Desktop at:
https://github.com/opsdroid/opsdroid-desktop/releases
INFO opsdroid: ========================================
DEBUG asyncio: Using selector: KqueueSelector
DEBUG opsdroid.loader: Loaded loader.
DEBUG opsdroid.loader: Loading modules from config...
WARNING opsdroid.loader: No databases in configuration. This will cause skills which store things in memory to lose data when opsdroid is restarted.
DEBUG opsdroid.loader: Loading parsers modules...
DEBUG opsdroid.loader: Loaded parsers: opsdroid.parsers.dialogflow.
DEBUG opsdroid.loader: Loading skill modules...
DEBUG opsdroid.loader: Updating dance...
DEBUG opsdroid.loader: b'Already up to date.'
DEBUG opsdroid.loader: b'Current branch master is up to date.'
DEBUG opsdroid.loader: Couldn't find the file requirements.txt, skipping.
DEBUG opsdroid.loader: Loaded skill: opsdroid-modules.skill.dance.
DEBUG opsdroid.loader: Updating hello...
DEBUG opsdroid.loader: b'Already up to date.'
DEBUG opsdroid.loader: b'Current branch master is up to date.'
DEBUG opsdroid.loader: Couldn't find the file requirements.txt, skipping.
DEBUG opsdroid.loader: Loaded skill: opsdroid-modules.skill.hello.
DEBUG opsdroid.loader: Updating loudnoises...
DEBUG opsdroid.loader: b'Already up to date.'
DEBUG opsdroid.loader: b'Current branch master is up to date.'
DEBUG opsdroid.loader: Couldn't find the file requirements.txt, skipping.
DEBUG opsdroid.loader: Loaded skill: opsdroid-modules.skill.loudnoises.
DEBUG opsdroid.loader: Updating seen...
DEBUG opsdroid.loader: b'Already up to date.'
DEBUG opsdroid.loader: b'Current branch master is up to date.'
DEBUG opsdroid.loader: b'Processing /Users/fabiorosado/Library/Caches/pip/wheels/c8/61/2f/47076152dc9487142c2ae48754c87539ff0993decf0fc2f198/ago-0.0.93-py3-none-any.whl'
DEBUG opsdroid.loader: b'Installing collected packages: ago'
DEBUG opsdroid.loader: b'Successfully installed ago-0.0.93'
DEBUG opsdroid.loader: b'WARNING: Target directory /Users/fabiorosado/Library/Application Support/opsdroid/site-packages/ago.py already exists. Specify --upgrade to force replacement.'
DEBUG opsdroid.loader: b'WARNING: Target directory /Users/fabiorosado/Library/Application Support/opsdroid/site-packages/__pycache__ already exists. Specify --upgrade to force replacement.'
DEBUG opsdroid.loader: b'WARNING: Target directory /Users/fabiorosado/Library/Application Support/opsdroid/site-packages/ago-0.0.93.dist-info already exists. Specify --upgrade to force replacement.'
DEBUG opsdroid.loader: Loaded skill: opsdroid-modules.skill.seen.
DEBUG opsdroid.loader: Loading connector modules...
DEBUG opsdroid.loader: Loaded connector: opsdroid.connector.websocket.
DEBUG opsdroid.loader: Loaded connector: opsdroid.connector.shell.
DEBUG opsdroid.core: Loaded 4 skills.
DEBUG opsdroid.connector.websocket: Starting Websocket connector.
DEBUG opsdroid.connector.shell: Loaded shell Connector.
DEBUG opsdroid.connector.shell: Connecting to shell.
INFO opsdroid.web: Started web server on http://0.0.0.0:8080
INFO opsdroid.core: Opsdroid is now running, press ctrl+c to exit.
opsdroid> DEBUG opsdroid.memory: Getting seen from memory.
DEBUG opsdroid.memory: Putting seen to memory.
hi
DEBUG opsdroid.core: Parsing input: <opsdroid.events.Message(text=hi)>.
DEBUG opsdroid.core: Processing parsers...
DEBUG opsdroid.memory: Getting seen from memory.
DEBUG opsdroid.memory: Putting seen to memory.
DEBUG asyncio: Using selector: KqueueSelector
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/fabiorosado/Documents/GitHub.tmp/opsdroid/opsdroid/__main__.py", line 12, in <module>
    init()
  File "/Users/fabiorosado/Documents/GitHub.tmp/opsdroid/opsdroid/__main__.py", line 9, in init
    opsdroid.cli.cli()
  File "/Users/fabiorosado/.local/share/virtualenvs/opsdroid-AmeH9qot/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/fabiorosado/.local/share/virtualenvs/opsdroid-AmeH9qot/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/fabiorosado/.local/share/virtualenvs/opsdroid-AmeH9qot/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/fabiorosado/.local/share/virtualenvs/opsdroid-AmeH9qot/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/fabiorosado/.local/share/virtualenvs/opsdroid-AmeH9qot/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/fabiorosado/Documents/GitHub.tmp/opsdroid/opsdroid/cli/start.py", line 42, in start
    opsdroid.run()
  File "/Users/fabiorosado/Documents/GitHub.tmp/opsdroid/opsdroid/core.py", line 165, in run
    self.eventloop.run_until_complete(asyncio.gather(*pending))
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 583, in run_until_complete
    return future.result()
  File "/Users/fabiorosado/Documents/GitHub.tmp/opsdroid/opsdroid/connector/shell/__init__.py", line 93, in _parse_message
    await self.parseloop()
  File "/Users/fabiorosado/Documents/GitHub.tmp/opsdroid/opsdroid/connector/shell/__init__.py", line 88, in parseloop
    await self.opsdroid.parse(message)
  File "/Users/fabiorosado/Documents/GitHub.tmp/opsdroid/opsdroid/core.py", line 498, in parse
    ranked_skills = await self.get_ranked_skills(unconstrained_skills, event)
  File "/Users/fabiorosado/Documents/GitHub.tmp/opsdroid/opsdroid/core.py", line 429, in get_ranked_skills
    if dialogflow and dialogflow["enabled"]:
KeyError: 'enabled'
ERROR: Unhandled exception in opsdroid, exiting...
ERROR: Unhandled exception in opsdroid, exiting...
```

</details>

## Versions
- **Opsdroid version:** opsdroid v0.8.1+833.g5c8dbb7.dirty.
- **Python version:** Python 3.8.1
- **OS/Docker version:** MacOs Catalina

## Configuration File
Please include your version of the configuration file below.

```yaml
parsers:
  dialogflow:
    project-id: test-ddd33
```

## Additional Details
Any other details you wish to include such as screenshots, console messages, etc.


<!-- Love opsdroid? Please consider supporting our collective:
 +👉  https://opencollective.com/opsdroid/donate -->



Discussion:
We should also check if the envvar `GOOGLE_APPLICATION_CREDENTIALS` exists and if not we should log it. Currently we are [raising a warning](https://github.com/opsdroid/opsdroid/blob/f69714045da88362587ffef10f3dabe928689bf3/opsdroid/parsers/dialogflow.py#L55) and opsdroid handles this error and doesn't send the warning message.

If that envvar doesn't exist the user should be warned about that but opsdroid should still run fine.
After digging further the issue seems to happen on every connector. I'm working on a fix right now

Resolving patch:
diff --git a/opsdroid/const.py b/opsdroid/const.py
--- a/opsdroid/const.py
+++ b/opsdroid/const.py
@@ -36,9 +36,6 @@
 
 LUISAI_DEFAULT_URL = "https://westus.api.cognitive.microsoft.com/luis/v2.0/apps/"
 
-DIALOGFLOW_API_ENDPOINT = "https://api.dialogflow.com/v1/query"
-DIALOGFLOW_API_VERSION = "20150910"
-
 WITAI_DEFAULT_VERSION = "20170307"
 WITAI_API_ENDPOINT = "https://api.wit.ai/message?"
 
diff --git a/opsdroid/core.py b/opsdroid/core.py
--- a/opsdroid/core.py
+++ b/opsdroid/core.py
@@ -31,6 +31,7 @@
 from opsdroid.parsers.watson import parse_watson
 from opsdroid.parsers.rasanlu import parse_rasanlu, train_rasanlu
 from opsdroid.parsers.crontab import parse_crontab
+from opsdroid.helper import get_parser_config
 
 
 _LOGGER = logging.getLogger(__name__)
@@ -315,9 +316,9 @@ async def train_parsers(self, skills):
             skills (list): A list of all the loaded skills.
 
         """
-        if "parsers" in self.config:
-            parsers = self.config["parsers"] or {}
-            rasanlu = parsers.get("rasanlu")
+        if "parsers" in self.modules:
+            parsers = self.modules.get("parsers", {})
+            rasanlu = get_parser_config("rasanlu", parsers)
             if rasanlu and rasanlu["enabled"]:
                 await train_rasanlu(rasanlu, skills)
 
@@ -448,38 +449,38 @@ async def get_ranked_skills(self, skills, message):
             ranked_skills += await parse_regex(self, skills, message)
             ranked_skills += await parse_format(self, skills, message)
 
-        if "parsers" in self.config:
+        if "parsers" in self.modules:
             _LOGGER.debug(_("Processing parsers..."))
-            parsers = self.config["parsers"] or {}
+            parsers = self.modules.get("parsers", {})
 
-            dialogflow = parsers.get("dialogflow")
+            dialogflow = get_parser_config("dialogflow", parsers)
             if dialogflow and dialogflow["enabled"]:
                 _LOGGER.debug(_("Checking dialogflow..."))
                 ranked_skills += await parse_dialogflow(
                     self, skills, message, dialogflow
                 )
 
-            luisai = parsers.get("luisai")
+            luisai = get_parser_config("luisai", parsers)
             if luisai and luisai["enabled"]:
                 _LOGGER.debug(_("Checking luisai..."))
                 ranked_skills += await parse_luisai(self, skills, message, luisai)
 
-            sapcai = parsers.get("sapcai")
+            sapcai = get_parser_config("sapcai", parsers)
             if sapcai and sapcai["enabled"]:
                 _LOGGER.debug(_("Checking SAPCAI..."))
                 ranked_skills += await parse_sapcai(self, skills, message, sapcai)
 
-            witai = parsers.get("witai")
+            witai = get_parser_config("witai", parsers)
             if witai and witai["enabled"]:
                 _LOGGER.debug(_("Checking wit.ai..."))
                 ranked_skills += await parse_witai(self, skills, message, witai)
 
-            watson = parsers.get("watson")
+            watson = get_parser_config("watson", parsers)
             if watson and watson["enabled"]:
                 _LOGGER.debug(_("Checking IBM Watson..."))
                 ranked_skills += await parse_watson(self, skills, message, watson)
 
-            rasanlu = parsers.get("rasanlu")
+            rasanlu = get_parser_config("rasanlu", parsers)
             if rasanlu and rasanlu["enabled"]:
                 _LOGGER.debug(_("Checking Rasa NLU..."))
                 ranked_skills += await parse_rasanlu(self, skills, message, rasanlu)
diff --git a/opsdroid/helper.py b/opsdroid/helper.py
--- a/opsdroid/helper.py
+++ b/opsdroid/helper.py
@@ -182,6 +182,28 @@ def add_skill_attributes(func):
     return func
 
 
+def get_parser_config(name, modules):
+    """Get parser from modules list.
+
+    After the change to the configuration we are adding the "enabled" flag to each
+    active module, this allows us to disable to module if there is any problem with
+    it. This helper method helps getting the config from the list of active parsers.
+
+    Args:
+        name (string): Name of the parser to be fetched.
+        modules (list): List of all active modules.
+
+    Returns:
+        dict or None: The module config or None if not found.
+
+    """
+    if modules:
+        for parser in modules:
+            if parser["config"]["name"] == name:
+                return parser["config"]
+    return None
+
+
 def get_config_option(options, config, found, not_found):
     """Get config details and return useful information to list active modules.
 


Test patch:
diff --git a/tests/test_core.py b/tests/test_core.py
--- a/tests/test_core.py
+++ b/tests/test_core.py
@@ -325,9 +325,18 @@ async def test_parse_regex_insensitive(self):
     async def test_parse_dialogflow(self):
         os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "path/test.json"
         with OpsDroid() as opsdroid:
-            opsdroid.config["parsers"] = {
-                "dialogflow": {"project-id": "test", "enabled": True}
+            opsdroid.modules = {
+                "parsers": [
+                    {
+                        "config": {
+                            "name": "dialogflow",
+                            "project-id": "test",
+                            "enabled": True,
+                        }
+                    }
+                ]
             }
+
             dialogflow_action = "smalltalk.greetings.whatsup"
             skill = amock.CoroutineMock()
             mock_connector = Connector({}, opsdroid=opsdroid)
@@ -349,7 +358,9 @@ async def test_parse_dialogflow(self):
 
     async def test_parse_luisai(self):
         with OpsDroid() as opsdroid:
-            opsdroid.config["parsers"] = {"luisai": {"enabled": True}}
+            opsdroid.modules = {
+                "parsers": [{"config": {"name": "luisai", "enabled": True}}]
+            }
             luisai_intent = ""
             skill = amock.CoroutineMock()
             mock_connector = Connector({}, opsdroid=opsdroid)
@@ -366,7 +377,11 @@ async def test_parse_luisai(self):
 
     async def test_parse_rasanlu(self):
         with OpsDroid() as opsdroid:
-            opsdroid.config["parsers"] = {"rasanlu": {"enabled": True}}
+            opsdroid.modules = {
+                "parsers": [
+                    {"config": {"name": "rasanlu", "module": "", "enabled": True}}
+                ]
+            }
             rasanlu_intent = ""
             skill = amock.CoroutineMock()
             mock_connector = Connector({}, opsdroid=opsdroid)
@@ -380,7 +395,9 @@ async def test_parse_rasanlu(self):
 
     async def test_parse_sapcai(self):
         with OpsDroid() as opsdroid:
-            opsdroid.config["parsers"] = {"sapcai": {"enabled": True}}
+            opsdroid.modules = {
+                "parsers": [{"config": {"name": "sapcai", "enabled": True}}]
+            }
             sapcai_intent = ""
             skill = amock.CoroutineMock()
             mock_connector = Connector({}, opsdroid=opsdroid)
@@ -394,7 +411,9 @@ async def test_parse_sapcai(self):
 
     async def test_parse_watson(self):
         with OpsDroid() as opsdroid:
-            opsdroid.config["parsers"] = {"watson": {"enabled": True}}
+            opsdroid.modules = {
+                "parsers": [{"config": {"name": "watson", "enabled": True}}]
+            }
             watson_intent = ""
             skill = amock.CoroutineMock()
             mock_connector = Connector({}, opsdroid=opsdroid)
@@ -406,7 +425,9 @@ async def test_parse_watson(self):
 
     async def test_parse_witai(self):
         with OpsDroid() as opsdroid:
-            opsdroid.config["parsers"] = {"witai": {"enabled": True}}
+            opsdroid.modules = {
+                "parsers": [{"config": {"name": "witai", "enabled": True}}]
+            }
             witai_intent = ""
             skill = amock.CoroutineMock()
             mock_connector = Connector({}, opsdroid=opsdroid)
@@ -526,7 +547,9 @@ async def test_start_databases(self):
 
     async def test_train_rasanlu(self):
         with OpsDroid() as opsdroid:
-            opsdroid.config["parsers"] = {"rasanlu": {"enabled": True}}
+            opsdroid.modules = {
+                "parsers": [{"config": {"name": "rasanlu", "enabled": True}}]
+            }
             with amock.patch("opsdroid.parsers.rasanlu.train_rasanlu"):
                 await opsdroid.train_parsers({})
 
diff --git a/tests/test_helper.py b/tests/test_helper.py
--- a/tests/test_helper.py
+++ b/tests/test_helper.py
@@ -4,6 +4,7 @@
 import tempfile
 import unittest
 import unittest.mock as mock
+import pytest
 
 from opsdroid.helper import (
     del_rw,
@@ -15,6 +16,7 @@
     JSONDecoder,
     convert_dictionary,
     get_config_option,
+    get_parser_config,
 )
 
 
@@ -167,3 +169,54 @@ def test_dict_to_datetime(self):
         decoder = JSONDecoder()
         obj = decoder(test_obj)
         self.assertEqual(datetime.datetime(2018, 10, 2, 0, 41, 17, 74644), obj)
+
+
+def test_get_parser_config():
+    parsers = [
+        {
+            "name": "dialogflow",
+            "module": "",
+            "config": {
+                "name": "dialogflow",
+                "module": "",
+                "project-id": "test-ddd33",
+                "type": "parsers",
+                "enabled": True,
+                "entrypoint": None,
+                "is_builtin": "",
+                "module_path": "opsdroid.parsers.dialogflow",
+                "install_path": "",
+                "branch": "master",
+            },
+            "intents": None,
+        },
+        {
+            "name": "regex",
+            "module": "",
+            "config": {
+                "name": "regex",
+                "module": "",
+                "type": "parsers",
+                "enabled": True,
+                "entrypoint": None,
+                "is_builtin": "",
+                "module_path": "opsdroid.parsers.regex",
+                "install_path": "",
+                "branch": "master",
+            },
+            "intents": None,
+        },
+    ]
+
+    dialogflow_config = get_parser_config("dialogflow", parsers)
+
+    assert dialogflow_config
+    assert dialogflow_config["name"] == "dialogflow"
+
+
+def test_get_parser_config_none():
+    parsers = {}
+
+    config = get_parser_config("dialogflow", parsers)
+
+    assert not config