Repository: opsdroid/opsdroid

Issue:
rasanlu parser : dict object has no attribute matchers
Hi all,

# Description
When using rasanlu parser with no intents file (no need to train the module, already done) an attribute issue error happened in function **_get_all_intents(skills)**

```
...
    matchers = [matcher for skill in skills for matcher in skill.matchers]
  File "/Users/IOBreaker/Developments/Bots/opsdroid/opsdroid/parsers/rasanlu.py", line 19, in <listcomp>
    matchers = [matcher for skill in skills for matcher in skill.matchers]
AttributeError: 'dict' object has no attribute 'matchers'
```
## Steps to Reproduce
1- create or use a skill that respond to an intent
2- use rasanlu as a parser
3- do not provide an intent file to opsdroid train your model outside of opsdroid
4- start opsdroid

## Expected Functionality
if no intent provided, not training step should be issued (no crash)


## Experienced Functionality

issue error happened in function **_get_all_intents(skills)**

```
INFO opsdroid.parsers.rasanlu: Starting Rasa NLU training.
DEBUG asyncio: Using selector: KqueueSelector
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/IOBreaker/Developments/Bots/opsdroid/opsdroid/__main__.py", line 206, in <module>
    init()
  File "/Users/IOBreaker/Developments/Bots/opsdroid/opsdroid/__main__.py", line 203, in init
    main()
  File "/Users/IOBreaker/Developments/Bots/opsdroid/venv/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/IOBreaker/Developments/Bots/opsdroid/venv/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/IOBreaker/Developments/Bots/opsdroid/venv/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/IOBreaker/Developments/Bots/opsdroid/venv/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/IOBreaker/Developments/Bots/opsdroid/opsdroid/__main__.py", line 196, in main
    opsdroid.load()
  File "/Users/IOBreaker/Developments/Bots/opsdroid/opsdroid/core.py", line 152, in load
    self.train_parsers(self.modules["skills"])
  File "/Users/IOBreaker/Developments/Bots/opsdroid/opsdroid/core.py", line 270, in train_parsers
    asyncio.gather(*tasks, loop=self.eventloop))
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 568, in run_until_complete
    return future.result()
  File "/Users/IOBreaker/Developments/Bots/opsdroid/opsdroid/parsers/rasanlu.py", line 87, in train_rasanlu
    intents = await _get_all_intents(skills)
  File "/Users/IOBreaker/Developments/Bots/opsdroid/opsdroid/parsers/rasanlu.py", line 19, in _get_all_intents
    matchers = [matcher for skill in skills for matcher in skill.matchers]
  File "/Users/IOBreaker/Developments/Bots/opsdroid/opsdroid/parsers/rasanlu.py", line 19, in <listcomp>
    matchers = [matcher for skill in skills for matcher in skill.matchers]
AttributeError: 'dict' object has no attribute 'matchers'
```

This error is due to this line :
> matchers = [matcher for skill in skills for matcher in skill.matchers]


Exeample of 3 skills used with rasanlu parser :

```
[
{'module': <module 'opsdroid-modules.skill.discussion' from '/Users/IOBreaker/Library/Application Support/opsdroid/opsdroid-modules/skill/discussion/__init__.py'>, 'config': {'name': 'discussion', 'path': '/Users/IOBreaker/Developments/Bots/opsdroid-data/skills/skill-rasanlu-discussion', 'debug': False, 'no-cache': True, 'module': '', 'type': 'skill', 'is_builtin': None, 'entrypoint': None, 'module_path': 'opsdroid-modules.skill.discussion', 'install_path': '/Users/IOBreaker/Library/Application Support/opsdroid/opsdroid-modules/skill/discussion', 'branch': 'master'}, 'intents': None}, 

{'module': <module 'opsdroid-modules.skill.hyperv' from '/Users/IOBreaker/Library/Application Support/opsdroid/opsdroid-modules/skill/hyperv/__init__.py'>, 'config': {'name': 'hyperv', 'path': '/Users/IOBreaker/Developments/Bots/opsdroid-data/skills/skill-rasanlu-hyperv', 'apixu-key': 'ee6eef66ee4e4ee3aee64539192002', 'debug': False, 'no-cache': True, 'module': '', 'type': 'skill', 'is_builtin': None, 'entrypoint': None, 'module_path': 'opsdroid-modules.skill.hyperv', 'install_path': '/Users/IOBreaker/Library/Application Support/opsdroid/opsdroid-modules/skill/hyperv', 'branch': 'master'}, 'intents': None},

 {'module': <module 'opsdroid-modules.skill.vmware' from '/Users/IOBreaker/Library/Application Support/opsdroid/opsdroid-modules/skill/vmware/__init__.py'>, 'config': {'name': 'vmware', 'path': '/Users/IOBreaker/Developments/Bots/opsdroid-data/skills/skill-vmware', 'no-cache': True, 'module': '', 'type': 'skill', 'is_builtin': None, 'entrypoint': None, 'module_path': 'opsdroid-modules.skill.vmware', 'install_path': '/Users/IOBreaker/Library/Application Support/opsdroid/opsdroid-modules/skill/vmware', 'branch': 'master'}, 'intents': None}
 ]
```

## Versions
- **Opsdroid version: v0.14.1+19.g6360912**
- **Python version: 3.7.0**


## Configuration File

```
parsers:
#  ## Rasanlu (http://opsdroid.readthedocs.io/en/stable/matchers/rasanlu)
  - name: rasanlu
    url: http://localhost:5000
    project: myProject
    model: myModule
    min-score: 0.8

skills:
  - name: social
    path: /Users/IOBreaker/Developments/Bots/opsdroid-data/skills/skill-rasanlu-vmware
    debug: true
    no-cache: true

```



Discussion:
Thanks for raising this! Looks like we need some more error checking!
Hi all,

can someone confirm why the code is passing each skill.matchers to get intents !?
```python
matchers = [matcher for skill in skills for matcher in skill.matchers] 
```

What i can notice is that intents are linked to the skill itself as you can see in this log

```
INFO opsdroid.parsers.rasanlu: Starting Rasa NLU training.
DEBUG opsdroid.parsers.rasanlu: =======> skill = {'module': <module 'opsdroid-modules.skill.thanks' from '/home/hicham/.local/share/opsdroid/opsdroid-modules/skill/thanks/__init__.py'>, 'config': {'name': 'thanks', 'path': '/home/hicham/Developments/Bots/opsdroid/miBot/skills/skill-thanks', 'no-cache': True, 'module': '', 'type': 'skill', 'is_builtin': None, 'entrypoint': None, 'module_path': 'opsdroid-modules.skill.thanks', 'install_path': '/home/hicham/.local/share/opsdroid/opsdroid-modules/skill/thanks', 'branch': 'master'}, 'intents': '## intent:thanks\n- thanks\n- thx\n- thank you'}
```

I am working on a fix (the fix is working and ready for deployment) but i want to be sure that intent has nothing to do with skill.matchers in our case because intents can be loaded directly from each skill

```python
    intents = [skill["intents"] for skill in skills
               if skill["intents"] is not None]
```

Regards
@jacobtomlinson or @FabioRosado 
Any idea please ?
We are using the RasaNLU format for intents and it is the only matcher which uses them. However I'm keen to use it as a generic way to specify intents and so it should be treated separately to the matchers.
No problem.
I am just trying to understand the logic befind the code :-)

the function **self._load_intents(config)** in **loader.py** is called at the beginning, when it find a intents.md file in the skill's directory, the content of this file is added to the intents dict.

This produce the bellow output :  **'intents': '## intent:user_say_iam_back\ni am back\ni am back again'**

```
INFO opsdroid.parsers.rasanlu: Starting Rasa NLU training.
DEBUG opsdroid.parsers.rasanlu: ======> RASA : {'module': <module 'opsdroid-modules.skill.social' from '/Users/hicham/Library/Application Support/opsdroid/opsdroid-modules/skill/social/__init__.py'>, 'config': {'name': 'social', 'path': '/Users/hicham/Developments/Bots/opsdroid-data/skills/skill-rasanlu-social', 'debug': False, 'no-cache': True, 'module': '', 'type': 'skill', 'is_builtin': None, 'entrypoint': None, 'module_path': 'opsdroid-modules.skill.social', 'install_path': '/Users/hicham/Library/Application Support/opsdroid/opsdroid-modules/skill/social', 'branch': 'master'}, 'intents': '## intent:user_say_iam_back\ni am back\ni am back again'}
```

as you can see, no matcher attribute or dict is added to the skill's dict.

the second step after is calling the function def **train_parsers(self, skills)** in **core.py**  to train rasanlu.

```python
async def _get_all_intents(skills):
    """Get all skill intents and concatenate into a single markdown string."""
    matchers = [matcher for skill in skills for matcher in skill.matchers]
    intents = [matcher["intents"] for matcher in matchers
               if matcher["intents"] is not None]
    if not intents:
        return None
    intents = "\n\n".join(intents)
    return unicodedata.normalize("NFKD", intents).encode('ascii')
```

In this step no matcher exist in skill's dict, so it is normal to have an error when executing

```python
matchers = [matcher for skill in skills for matcher in skill.matchers]
```

It's why is suggested to use instead

```python
intents = [skill["intents"] for skill in skills
               if skill["intents"] is not None]
```

If matchers info must be in the skill dict before calling the **train_parsers(self, skills)** and **_get_all_intents(skills)** it means that somewhere a step is missing !!




Yeah thanks for catching this. If you could make your proposed changes that would be great.
ok, I will work on it and submit a PR when it's done
Unfortunately, this issue is purely not the only one, it is a drop in the ocean.

Correcting the initial issue i am now getting this one :

```
ERROR opsdroid.parsers.rasanlu: Bad Rasa NLU response - {
  "error": "Content-Type must be 'application/x-yml' or 'application/json'"
}
```

When trying to to correct this second issue,  got an other one

```
ERROR opsdroid.parsers.rasanlu: Bad Rasa NLU response - {
  "error": "'str' object has no attribute 'get'"
}

'status': 400, 'reason': 'Bad Request'

```

So 3 issues in One

Continue working :-)
Fixed the two first errors.
the error with str object is related to rasanlu itself i think

here is my testing version of rasanlu

```
{
  "version": "0.14.4",
  "minimum_compatible_version": "0.13.0a2"
}
```


I added a new **train** parameter in config file to let users choose if they want opsdroid to train the model or not _**(training is not always needed)**_ :

```yaml
parsers:
#  ## Rasanlu (http://opsdroid.readthedocs.io/en/stable/matchers/rasanlu)
  - name: rasanlu
    url: http://localhost:5000
    project: ergo
    model: ergo
    token: "zgazrgzrgzrgazrgaz4er5et3gezg35a"
    train: true.  # <------
    min-score: 0.8
```

Is it ok  for you ?
I seem to remember that the training should only happen if the intents have changed. Therefore the `train` config option may be unnecessary.
hi @jacobtomlinson , understood but what i meean is that the training step here is always started even if the rasa training does not occurs. why starting it each time even if it's not needed (or user doses not want it to happen).
The only way to prevent this for the moment is to force user to not have/or move the md file out of the skill directory

No problem to get `train` out of the scope if you think it's is not a relevant change 
I think if there is an `md` file then the training step should be triggered. It should then take a hash of the `md` file, compare it with the last trained hash (I can't remember where this is stored) and decide whether or not to actually do the training.

Therefore I think that this is intended behavior and shouldn't add overhead if training has already been done.
got it :-), thanks a lot

Resolving patch:
diff --git a/opsdroid/loader.py b/opsdroid/loader.py
--- a/opsdroid/loader.py
+++ b/opsdroid/loader.py
@@ -188,7 +188,7 @@ def _communicate_process(process):
 
     @staticmethod
     def _load_intents(config):
-        intent_file = os.path.join(config["install_path"], "intents.md")
+        intent_file = os.path.join(config["install_path"], "intents.yml")
         if os.path.isfile(intent_file):
             with open(intent_file, 'r') as intent_file_handle:
                 intents = intent_file_handle.read()
diff --git a/opsdroid/parsers/rasanlu.py b/opsdroid/parsers/rasanlu.py
--- a/opsdroid/parsers/rasanlu.py
+++ b/opsdroid/parsers/rasanlu.py
@@ -16,9 +16,8 @@
 
 async def _get_all_intents(skills):
     """Get all skill intents and concatenate into a single markdown string."""
-    matchers = [matcher for skill in skills for matcher in skill.matchers]
-    intents = [matcher["intents"] for matcher in matchers
-               if matcher["intents"] is not None]
+    intents = [skill["intents"] for skill in skills
+               if skill["intents"] is not None]
     if not intents:
         return None
     intents = "\n\n".join(intents)
@@ -100,9 +99,16 @@ async def train_rasanlu(config, skills):
 
         url = await _build_training_url(config)
 
+        # https://github.com/RasaHQ/rasa_nlu/blob/master/docs/http.rst#post-train
+        # Note : The request should always be sent as
+        # application/x-yml regardless of wether you use
+        # json or md for the data format. Do not send json as
+        # application/json for example.+
+        headers = {'content-type': 'application/x-yml'}
+
         try:
             training_start = arrow.now()
-            resp = await session.post(url, data=intents)
+            resp = await session.post(url, data=intents, headers=headers)
         except aiohttp.client_exceptions.ClientConnectorError:
             _LOGGER.error(_("Unable to connect to Rasa NLU, training failed."))
             return False


Test patch:
diff --git a/tests/test_loader.py b/tests/test_loader.py
--- a/tests/test_loader.py
+++ b/tests/test_loader.py
@@ -217,7 +217,7 @@ def test_loading_intents(self):
             self._tmp_dir, os.path.normpath("test/module/test"))
         os.makedirs(config['install_path'], mode=0o777)
         intent_contents = "Hello world"
-        intents_file = os.path.join(config['install_path'], "intents.md")
+        intents_file = os.path.join(config['install_path'], "intents.yml")
         with open(intents_file, 'w') as intents:
             intents.write(intent_contents)
         loaded_intents = ld.Loader._load_intents(config)
diff --git a/tests/test_parser_rasanlu.py b/tests/test_parser_rasanlu.py
--- a/tests/test_parser_rasanlu.py
+++ b/tests/test_parser_rasanlu.py
@@ -346,9 +346,9 @@ async def test__get_all_intents(self):
             await self.getMockSkill(),
             await self.getMockSkill()
         ]
-        skills[0].matchers = [{"intents": "Hello"}]
-        skills[1].matchers = [{"intents": None}]
-        skills[2].matchers = [{"intents": "World"}]
+        skills[0] = {"intents": "Hello"}
+        skills[1] = {"intents": None}
+        skills[2] = {"intents": "World"}
         intents = await rasanlu._get_all_intents(skills)
         self.assertEqual(type(intents), type(b""))
         self.assertEqual(intents, b"Hello\n\nWorld")