Repository: opsdroid/opsdroid

Issue:
Change the way opsdroid prioritize regex / NLU skills
# Description
Actually opsdroid use a score system to decide what skill have to run when more than one skills match a message.
This score system is designed to prioritize NLU engines over regex (see #311).

The problem comes when you're working at the same time with some NLU engine and some regex skills. If the regex skill have some words of some NLU intent, it will be never executed, even though you write it exactly as in the regex.

## Steps to Reproduce
For example, you add a greetings intent to some NLU training with various sentences like 'hello', 'hello opsdroid', 'hi', 'hello there', 'whats up', 'good morning'...
And you have the min-score at 0.6.

Then you have a regex matcher with something like 'hello from (?P\w+)!'.

If a user writes "hello from Barcelona!" opsdroid will prioritize the NLU response, because it's almost secure that the NLU platform will detect the intent into the sentence.

## Expected Functionality
If a developer has the regex 'hello from (?P\w+)!' and it matches, it has to be prioritized over the NLU intent, because regex it's more specific.

## Proposed solution
My proposal is redesign the priority system by specificity. So I will prioritize skills in that order:
_parse (#450) > regex > NLU engines_

And, to decide what skill to execute if more than one match in the same level:
- In parse and regex, use the expression length. Longer expression normally it's more specific.
- In NLU engines, use the engine score (like now)

Any thought or discussion will be welcome :)


Discussion:
I agree that regex skills should be able to score higher than NLU skills if it truly is a better match. 

When writing this I was worried about simple regex's winning over advanced NLU, for example if an NLU engine is trained with sentences like "will it rain in {place} at {time}" and there is a regex that matches "will it rain" then I want the NLU to win.

But your examples demonstrate that the regex scores should definitely be able to beat the NLU score.

I disagree that parse and regex have higher priority over NLU, however I do agree that we should think about how to properly score them so things work better.

Perhaps a good approach would be a combination of message length, regex/parse length and the difference between the two.
> When writing this I was worried about simple regex's winning over advanced NLU, for example if an NLU engine is trained with sentences like "will it rain in {place} at {time}" and there is a regex that matches "will it rain" then I want the NLU to win.

And what about make win any regex with anchors (`^$`)? So if instead of "will it rain" the skill matches "^will it rain$", then you want this specific sentence and it must win NLU engines.
I have another idea. Add an optional parameter to the regex matcher, so the programmer can change the priority to be above or below the NLU skills. Then also create a configuration option that sets the default regex priority, so the operator can decide if in general if they want regex to have priority, or not.
I like @go8ose idea because you give the power to the user to choose NLU or Regex priority. 
One thing that I would like to mention is that in some cases the NLU will give a 100% score when detecting the intent of a sentence, so we shouldn't use the score to specify which parser should have priority. Perhaps using a sort of flag that points either to regex or the NLU? 
@go8ose I like this idea of changing the weighting of regex in the configuration. 

@FabioRosado I'm not sure I understand what you mean. If something has a 100% score then I think that should always win. Or do you mean if two parsers both return 100%?
Sorry I was still half asleep when I wrote that haha
I mean we should use something other than the score to decide which parser has priority because in some cases the NLU will return a 100% score on the sentence and that would make the regex be skipped again
But if the NLU returns 100% then surely it is 100% confident of the match? The whole point of the scores is to tell opsdroid which one to run. 

Ideally I want to treat regex like any other parser. It matches a string, it calculates a score to say how confident it is on the match and then opsdroid uses the scores to decide which parser to use.
@jacobtomlinson I didn't think regex's did partial matches. They either match, or they don't. I mean, in a regex with optional elements you might have some groups that have nothing in them, but the regex would still have matched or not.

If an NLU is 100% sure it's a match for the intent, but a regex is also 100% sure, how do you break the tie? If I had an existing opsdroid bot, and I add as skill to it which has a regex, and then it doesn't fire, I'm going to be confused. But if I have a bot with multiple skills using NLU, the chance of a tie seems much lower, and you'd probably understand that you might need to "train" your NLU more. if that is what you do with NLUs (1).

As an example, consider where an operator wants to make available a single command line program to the bot. They might have something like:

```python
@match_regex(r'ping ([0-9]{1,3}\.){3}[0-9]{1,3}$')
async def ping(opsdroid, config, message):
    mac = message.regex.group(1)

    # Fire off the command "ping -c 4 {mac}"
    # TODO: work out how to do this in async world, i.e. https://docs.python.org/3/library/asyncio-subprocess.html

    # Send the output
    await message.respond(text)
```

In my mind, this is a perfectly normal thing to expect to be able to do in the chatops world. If an NLU thinks that "ping 127.0.0.1" should mean to reply with "there is no place like home", then I'm going to be confused (2). Maybe my focus is on the chatops world, but Jacob is thinking of other areas where you don't want your bot doing low level systems tasks.

So my proposal is
 a) NLU return confidence scores of 0 to 100
 b) regex matchers have an optional parameter so the programmer can express their confidence, in the range of 0 to 150
 c) in configuration you have an option to set the default regex confidence if none is expressed, in the range of 0 to 150. The default for this configuration is 50 (3)

(1) It should be pretty clear by now I have almost no experience with NLU systems.

(2) See (1), I don't know why it would do that.

(3) I don't know what the numbers should be. However with the numbers I've given I'm trying to demonstrate that I think 50 meets with what Jacob is thinking. In any bot I deploy I'd probably set it to 150.
@go8ose Thanks for that, really useful stuff!

To break ties the parsers have an order of priority in the code (due to the nature of the code, not intentionally), I guess what @FabioRosado was suggesting is the ability to reorder them in the config. Regex is currently first if that makes a difference 😄.

In my head a partial regex match in your example above would be `please could you ping 8.8.8.8`, which I feel should get a lower score than `ping 8.8.8.8`. A failed example would be `please ping the ip 8.8.8.8`, as it wouldn't match.

If I trained an NLU model with examples like `please ping {ip}`, `ping {ip}`, `ping the ip {ip}`, etc then I would expect it to be more flexible than the regex and therefore be able to score higher. 

I see your point that an NLU parser with some kind of smalltalk skill enabled could end up hijacking the conversation a lot of the time, and so the regex skills should get high scores. 

My initial reaction is to give regex skills a scored based on how well it matched (using length, empty capture groups, etc) but distribute that score normally. The bounds of that distribution could be configurable (scores are currently probabilities and so can't go over 1). To ensure regex always wins you could configure the mean to 1 and the variance to 0. Or for situations where you want NLU to have a chance you could set the mean to 0.75 and the variance to 0.25, resulting in scores between 0.5 and 1 for regex.

I'm also totally open to the fact that I'm over thinking this. I'm more interested in the views and use cases of the community, which so far are compelling me to just default the score of a regex to 1 and allow configuration to another value if you want.
Also this might be useful

```python
  from asyncio.subprocess import PIPE
  import asyncio

  @match_regex(r"ping (?P<ip>([0-9]{1,3}\.){3}[0-9]{1,3})$")
  async def ping(opsdroid, config, message):
      ip = message.regex.group('ip')
      proc = await asyncio.create_subprocess_shell("ping -c 4 {ip}".format(ip=ip), stdin=None, stderr=None, stdout=PIPE)
      output = await proc.stdout.read()
      await message.respond(output.decode('utf-8'))
```
@jacobtomlinson  Yeah that was what I was trying to say, but you had it covered so its all good haha
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Resolving patch:
diff --git a/opsdroid/const.py b/opsdroid/const.py
--- a/opsdroid/const.py
+++ b/opsdroid/const.py
@@ -21,7 +21,7 @@
 LOCALE_DIR = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'locale')
 EXAMPLE_CONFIG_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)),
                                    "configuration/example_configuration.yaml")
-REGEX_MAX_SCORE = 0.6
+REGEX_SCORE_FACTOR = 0.6
 
 RASANLU_DEFAULT_URL = "http://localhost:5000"
 RASANLU_DEFAULT_PROJECT = "opsdroid"
diff --git a/opsdroid/matchers.py b/opsdroid/matchers.py
--- a/opsdroid/matchers.py
+++ b/opsdroid/matchers.py
@@ -2,6 +2,7 @@
 
 import logging
 
+from opsdroid.const import REGEX_SCORE_FACTOR
 from opsdroid.helper import get_opsdroid
 from opsdroid.web import Web
 
@@ -9,7 +10,7 @@
 _LOGGER = logging.getLogger(__name__)
 
 
-def match_regex(regex, case_sensitive=True):
+def match_regex(regex, case_sensitive=True, score_factor=None):
     """Return regex match decorator."""
     def matcher(func):
         """Add decorated function to skills list for regex matching."""
@@ -18,7 +19,8 @@ def matcher(func):
             config = opsdroid.loader.current_import_config
             regex_setup = {
                 "expression": regex,
-                "case_sensitive": case_sensitive
+                "case_sensitive": case_sensitive,
+                "score_factor": score_factor or REGEX_SCORE_FACTOR,
             }
             opsdroid.skills.append({"regex": regex_setup,
                                     "skill": func,
diff --git a/opsdroid/parsers/regex.py b/opsdroid/parsers/regex.py
--- a/opsdroid/parsers/regex.py
+++ b/opsdroid/parsers/regex.py
@@ -3,17 +3,14 @@
 import logging
 import re
 
-from opsdroid.const import REGEX_MAX_SCORE
-
-
 _LOGGER = logging.getLogger(__name__)
 
 
-async def calculate_score(regex):
+async def calculate_score(regex, score_factor):
     """Calculate the score of a regex."""
     # The score asymptotically approaches the max score
     # based on the length of the expression.
-    return (1 - (1 / ((len(regex) + 1) ** 2))) * REGEX_MAX_SCORE
+    return (1 - (1 / ((len(regex) + 1) ** 2))) * score_factor
 
 
 async def parse_regex(opsdroid, message):
@@ -21,17 +18,18 @@ async def parse_regex(opsdroid, message):
     matched_skills = []
     for skill in opsdroid.skills:
         if "regex" in skill:
-            if skill["regex"]["case_sensitive"]:
-                regex = re.search(skill["regex"]["expression"],
+            opts = skill["regex"]
+            if opts["case_sensitive"]:
+                regex = re.search(opts["expression"],
                                   message.text)
             else:
-                regex = re.search(skill["regex"]["expression"],
+                regex = re.search(opts["expression"],
                                   message.text, re.IGNORECASE)
             if regex:
                 message.regex = regex
                 matched_skills.append({
                     "score": await calculate_score(
-                        skill["regex"]["expression"]),
+                        opts["expression"], opts["score_factor"]),
                     "skill": skill["skill"],
                     "config": skill["config"],
                     "message": message


Test patch:
diff --git a/tests/test_parser_regex.py b/tests/test_parser_regex.py
--- a/tests/test_parser_regex.py
+++ b/tests/test_parser_regex.py
@@ -22,6 +22,22 @@ async def test_parse_regex(self):
             skills = await parse_regex(opsdroid, message)
             self.assertEqual(mock_skill, skills[0]["skill"])
 
+    async def test_parse_regex_priority(self):
+        with OpsDroid() as opsdroid:
+            regex = r"(.*)"
+
+            mock_skill_low = amock.CoroutineMock()
+            match_regex(regex, score_factor=0.6)(mock_skill_low)
+
+            mock_skill_high = amock.CoroutineMock()
+            match_regex(regex, score_factor=1)(mock_skill_high)
+
+            mock_connector = amock.CoroutineMock()
+            message = Message("Hello world", "user", "default", mock_connector)
+
+            skills = await opsdroid.get_ranked_skills(message)
+            self.assertEqual(mock_skill_high, skills[0]["skill"])
+
     async def test_parse_regex_raises(self):
         with OpsDroid() as opsdroid:
             mock_skill = amock.CoroutineMock()