Repository: MycroftAI/mycroft-core

Issue:
BUG: normalize_en fails on 'half'
Working on https://github.com/MycroftAI/mycroft-core/issues/1959, and got tripped up by this:

```
>>> normalize_en("three", True)
'3'
>>> normalize_en("three and a half", True)
'three and half'
```

The presence of the word 'half' seems to trip up the parser for every number that comes before it:

```
>>> normalize_en("three ten five eight", True)
'3 10 5 8'
>>> normalize_en("three ten five eight half", True)
'three ten five eight half'
>>> normalize_en("three half ten five eight", True)
'three half 10 5 8'
>>> normalize_en("three ten half five eight", True)
'three ten half 5 8'
```

Looks like there's a TODO to resolve this [in the code](https://github.com/MycroftAI/mycroft-core/blob/dev/mycroft/util/lang/parse_en.py#L1155), but no issue to track it.


Discussion:
It looks like [this commit](https://github.com/MycroftAI/mycroft-core/commit/60d29056783fbe2b7a1741865ed99ed825ae2e03) by @JarbasAI added that behavior to normalize().  I think we should remove that number replacement in normalize()  -- it is too dangerous to use extract_number() in that function.  I think we revert back to the simplified number replacement from before that commit.

Thoughts?
Removing it now would be an API break, which I feel like should be avoided unless strictly necessary (at the very least, we'd need to wait for a major version bump).

I also think the behavior (replacing numbers within the string) is useful. If we did pull it out in the next major version, it would probably be worth moving it to a second function. But this begs the question - if we were only to shift the danger to another function, is it worth going through the API break?

It might then be best to try to extract pieces of this function, and those that back it, into lots of discrete little private functions, and unit test everything extensively. If successful, we could avoid the API break, and effectively eliminate the dangers.
The change adding the "half" issue has been done during this release so we don't have any existing promise that the normalize shall convert numbers using the fully fledged extract_number method. (basically this feature extended the previous behavior)

I do however think it's probably better to have a full implementation rather than the old limited version in normalize. At least I think both extract_number and normalize should use the same function to do the basic number replace so we don't duplicate the code too much.
wouldn't solve that TODO fix this issue? i opened https://github.com/MycroftAI/mycroft-core/issues/1875 planing to use it here, just didn't have the time to tackle it yet

refactoring it into smaller pieces seems the way to go, and as @ChristopherRogers1991 says add unittests for all the pieces
I'm not opposed to a solid fix, but if it still has a TODO then I think a rollback of this new feature is a better immediate solution.  There are a lot of tricky situations when you are doing this at the topmost level before intent parsing, e.g.:
```
Set an alarm for half past five in the morning
```
Is awfully difficult to replace with numbers cleanly.  I honestly haven't attempted to debug this at all, I'm just reacting to the reported symptoms and what I've encountered in the past.

To be clear, I think simple replacements could be ok, such as:
```
Set an alarm for half past 5 in the morning
```

But I think time parsing would become difficult if it has to deal with a system that gives it:
```
Set and alarm for 0.5 past 5 in the morning
```
Or some such.
I think the first step is probably going to be to define what `normalize` means, and once defined, I don't think we should change it. If we decide that normalizing includes replacing words with numbers, then we have to treat this as a bug and try to fix it as quickly as possible. If normalizing does not include that feature, we roll it back and (potentially) create a separate function for the number parsing.

As hard as it may be for an alarm skill author to deal with `set an alarm for 0.5 past 5 in the morning`, it's significantly more challenging if they're not sure if they'll get that or `set and alarm for half past 5 in the morning` or `Set and alarm for half past five in the morning` or something else - a changing definition is worse bad definition, so we've got to pick something and stick to it.

I would personally consider the change to start parsing the numbers an API break, since it changed the expected output from the function. Given the recency with which this change was made (didn't realize how recent it was when I commented before), I'm in favor of the roll back.

I maintain that the functionality of replacing the text with numbers is generally useful, and we should implement it as a secondary function (or set of functions - may be useful to have a couple for different purposes, e.g. one that does standard ints, floats, etc, and another to deal with time related language)..
Worth noting is that there are several language specific `normalize` functions, all wrapped/called by the main normalize function. We probably also need to make sure they're all relatively consistent. Otherwise I expect it'll be challenging to have language agnostic skills (the logic would need to change based on the language of the user)..
i think this is a bug, and rolling back won't really solve this, because before it replaced all numbers 1 to 20 only, does not make sense that "one hundred" become "1 hundred", and "thirty times twenty" becomes "thirty times 20"

rolling back is just getting another poor implementation in place... i either expect all or none of the numbers to be converted

this is either a bug, or normalize should not handle numbers at all, there are already methods to extract numbers that the developers can use
in a way the TODO was already there but undocumented

previous implementation was even more buggy, "twenty two" becomes "20 2"
I intended normalize() originally to make intent parsing easier.  So extracting date, time, etc wouldn't be done at intent matching but rather at intent handling time.

That is why I only had it deal with low integers, which might be used in simple intent rules.  Things like 20.4 aren't very likely in an intent rule, but 3 is likely in a simple menu.

Extracting more complex numbers should be fine after the context of the intent is understood.  For example a phrase about alarms would have you in the context of time, so running extract_date() would be most appropriate rather and extracting a regular number with no thought about time idioms.

Advanced parsing should probably be done on the raw utterance, too.  Not normalized.

That was my intention.
Let's roll this back for now, then discuss for potential changes in 19.02
To be clear, I just want to back the changes that [this commit](https://github.com/MycroftAI/mycroft-core/commit/60d29056783fbe2b7a1741865ed99ed825ae2e03) made to the normalize() results.  I'm fine with the remainder of the things it added.

Resolving patch:
diff --git a/mycroft/util/lang/parse_en.py b/mycroft/util/lang/parse_en.py
--- a/mycroft/util/lang/parse_en.py
+++ b/mycroft/util/lang/parse_en.py
@@ -1137,21 +1137,15 @@ def normalize_en(text, remove_articles):
                          "you are not", "you are not", "you are", "you have"]
             word = expansion[contraction.index(word)]
 
-        normalized += " " + word
+        # Convert numbers into digits, e.g. "two" -> "2"
+        textNumbers = ["zero", "one", "two", "three", "four", "five", "six",
+                       "seven", "eight", "nine", "ten", "eleven", "twelve",
+                       "thirteen", "fourteen", "fifteen", "sixteen",
+                       "seventeen", "eighteen", "nineteen", "twenty"]
+
+        if word in textNumbers:
+            word = str(textNumbers.index(word))
 
-    # replace extracted numbers
-    numbers = extract_numbers_en(normalized)
-    # sort by string size, "twenty two" should be replaced before "two"
-    numbers.sort(key=lambda s: len(pronounce_number_en(s)), reverse=True)
-    for n in numbers:
-        txt = pronounce_number_en(n)
-        n = str(n)
-        if n.endswith(".0"):
-            n = n[:-2]
-        normalized = normalized.replace(txt, n)
-        # prnounced may be different from txt, ie
-        # pronounce(0.5) != half
-        # extract(half) == 0.5
-        # TODO account for this
+        normalized += " " + word
 
     return normalized[1:]  # strip the initial space


Test patch:
diff --git a/test/unittests/util/test_parse.py b/test/unittests/util/test_parse.py
--- a/test/unittests/util/test_parse.py
+++ b/test/unittests/util/test_parse.py
@@ -491,11 +491,15 @@ def test_numbers(self):
         self.assertEqual(normalize("that's eighteen nineteen twenty"),
                          "that is 18 19 20")
         self.assertEqual(normalize("that's one nineteen twenty two"),
-                         "that is 1 19 22")
+                         "that is 1 19 20 2")
         self.assertEqual(normalize("that's one hundred"),
-                         "that is 100")
+                         "that is 1 hundred")
         self.assertEqual(normalize("that's one two twenty two"),
-                         "that is 1 2 22")
+                         "that is 1 2 20 2")
+        self.assertEqual(normalize("that's one and a half"),
+                         "that is 1 and half")
+        self.assertEqual(normalize("that's one and a half and five six"),
+                         "that is 1 and half and 5 6")
 
     def test_multiple_numbers(self):
         self.assertEqual(extract_numbers("this is a one two three  test"),