internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: MycroftAI/mycroft-core

Issue:
Allow comments in dialog and voc files
Having a way to comment dialog and voc files would ease the translation process as the original developer can provide context useful for the translator, and the translator can explain things that motivate his/her choice of words.

I am thinking for instance of voc files where similar words to the target word are used to trigger the skill in order to workaround speech to text errors. Documenting that the meaning of those words should not be translated in a comment would be practical.

Example: the "high" word triggers the hello world skill because "high" is similar to "hi". In the Spanish translation we have "hola" (for "hello") and we should have "ola" (literally "wave") because it sounds like "hola"

Other file formats usually used for translation purposes allow developer/translator comments (see the .po file format for instance)


Discussion:
This is a really good idea and shouldn't be too hard to implement.

Adding checks in [mycroft/skills/skill_data.pyL40](https://github.com/MycroftAI/mycroft-core/blob/dev/mycroft/skills/skill_data.py#L40) for .voc files and [mycroft/skills/skill_data.pyL62](https://github.com/MycroftAI/mycroft-core/blob/dev/mycroft/skills/skill_data.py#L62) for regex files.

If the line starts with a `#` it should be ignored.

Resolving patch:
diff --git a/mycroft/skills/skill_data.py b/mycroft/skills/skill_data.py
--- a/mycroft/skills/skill_data.py
+++ b/mycroft/skills/skill_data.py
@@ -37,6 +37,8 @@ def load_vocab_from_file(path, vocab_type, emitter):
     if path.endswith('.voc'):
         with open(path, 'r') as voc_file:
             for line in voc_file.readlines():
+                if line.startswith("#"):
+                    continue
                 parts = line.strip().split("|")
                 entity = parts[0]
                 emitter.emit(Message("register_vocab", {
@@ -59,6 +61,8 @@ def load_regex_from_file(path, emitter, skill_id):
     if path.endswith('.rx'):
         with open(path, 'r') as reg_file:
             for line in reg_file.readlines():
+                if line.startswith("#"):
+                    continue
                 re.compile(munge_regex(line.strip(), skill_id))
                 emitter.emit(
                     Message("register_vocab",


Test patch:
Repository: MycroftAI/mycroft-core

Issue:
"signals" outlive the processes responsible for them
Working with a git clone development install, I couldn't get the Pairing code to put in the web interface.

I had to LOG debug the pairing skill to find out that it was stuck waiting for the "isSpeaking" "signal" to clear. I dug into the "signals" and found they were just files written to _/tmp/mycroft/ipc/signal._

Why didn't "isSpeaking" clear?  I can only guess it was an earlier _stop-mycroft.sh_  with unlucky timing.  After _stop-mycroft.sh_, **the file was still there.**  I removed the file and it worked.

Hours were lost.


Discussion:
Good find.  It should be pretty easy for someone to wipe out that signal file -- basically just check_for_signal("isSpeaking") -- in the Audio service startup.

Resolving patch:
diff --git a/mycroft/audio/main.py b/mycroft/audio/main.py
--- a/mycroft/audio/main.py
+++ b/mycroft/audio/main.py
@@ -30,7 +30,7 @@
 from mycroft.messagebus.client.ws import WebsocketClient
 from mycroft.messagebus.message import Message
 from mycroft.util import reset_sigint_handler, wait_for_exit_signal, \
-    create_daemon, create_echo_function
+    create_daemon, create_echo_function, check_for_signal
 from mycroft.util.log import LOG
 
 try:
@@ -466,6 +466,7 @@ def shutdown(self):
 def main():
     """ Main function. Run when file is invoked. """
     reset_sigint_handler()
+    check_for_signal("isSpeaking")
     ws = WebsocketClient()
     Configuration.init(ws)
     speech.init(ws)


Test patch:
Repository: MycroftAI/mycroft-core

Issue:
Gracefully fail when there is no localisation of an intent/skill
At the moment when Mycroft is started and you are starting to localise it, if you have only localised one skill all of the others cause Python errors/exceptions. This makes it more difficult to debug. Ideally these exceptions should be caught and the skill should be disabled with an informative error message.

![img](https://user-images.githubusercontent.com/449545/48378624-9066a700-e6d1-11e8-969a-ca2d00075f23.png)



Discussion:
I think it's doing basically what you're asking. It could however in these cases skip logging the entire stack trace of the error and instead just report that a missing file is blocking the skill from loading.

For the padatious intents this should be relatively easy to add.

Replace the `raise ValueError` in [register_intent_file](https://github.com/MycroftAI/mycroft-core/blob/dev/mycroft/skills/core.py#L850) and [register_entity_file](https://github.com/MycroftAI/mycroft-core/blob/dev/mycroft/skills/core.py#L882) with `raise FileNotFoundError`.

Then add an exception handler before the general one in [load_skill](https://github.com/MycroftAI/mycroft-core/blob/dev/mycroft/skills/core.py#L158)

```python
    except FileNotFoundError as e:
        LOG.error('Failed to load {} due to a missing file: {}'.format(name, str(e))
```
It could also check the extension of the file and leave a message like "You need to add a .voc file which is used by the intent parser" or something like that. 

Resolving patch:
diff --git a/mycroft/skills/core.py b/mycroft/skills/core.py
--- a/mycroft/skills/core.py
+++ b/mycroft/skills/core.py
@@ -155,6 +155,10 @@ def load_skill(skill_descriptor, bus, skill_id, BLACKLISTED_SKILLS=None):
             return skill
         else:
             LOG.warning("Module {} does not appear to be skill".format(name))
+    except FileNotFoundError as e:
+        LOG.error(
+            'Failed to load {} due to a missing file: {}'.format(name, str(e))
+            )
     except Exception:
         LOG.exception("Failed to load skill: " + name)
     return None
@@ -989,7 +993,9 @@ def register_intent_file(self, intent_file, handler):
 
         filename = self.find_resource(intent_file, 'vocab')
         if not filename:
-            raise ValueError('Unable to find "' + str(intent_file) + '"')
+            raise FileNotFoundError(
+                'Unable to find "' + str(intent_file) + '"'
+                )
 
         data = {
             "file_name": filename,
@@ -1021,7 +1027,9 @@ def register_entity_file(self, entity_file):
 
         filename = self.find_resource(entity_file + ".entity", 'vocab')
         if not filename:
-            raise ValueError('Unable to find "' + entity_file + '.entity"')
+            raise FileNotFoundError(
+                'Unable to find "' + entity_file + '.entity"'
+                )
         name = str(self.skill_id) + ':' + entity_file
 
         self.bus.emit(Message("padatious:register_entity", {


Test patch:
Repository: MycroftAI/mycroft-core

Issue:
Audio service crash
Running Mycroft Core version 0.8.3, the audio service crashed at some point when it was set to Google TTS. Here is the snippet of the traceback I was able to get before it overwrote the file when restarting:
```python
    speech.init(ws)
  File "/usr/local/lib/python2.7/site-packages/mycroft_core-18.2.3-py2.7.egg/mycroft/audio/speech.py", line 148, in init
    tts = TTSFactory.create()
  File "/usr/local/lib/python2.7/site-packages/mycroft_core-18.2.3-py2.7.egg/mycroft/tts/__init__.py", line 426, in create
    tts.validator.validate()
  File "/usr/local/lib/python2.7/site-packages/mycroft_core-18.2.3-py2.7.egg/mycroft/tts/__init__.py", line 347, in validate
    self.validate_connection()
  File "/usr/local/lib/python2.7/site-packages/mycroft_core-18.2.3-py2.7.egg/mycroft/tts/google_tts.py", line 44, in validate_connection
    'GoogleTTS server could not be verified. Please check your '
Exception: GoogleTTS server could not be verified. Please check your internet connection.
```
We should probably fall back to mimic if the TTS factory raises an exception.


Discussion:

Resolving patch:
diff --git a/mycroft/tts/__init__.py b/mycroft/tts/__init__.py
--- a/mycroft/tts/__init__.py
+++ b/mycroft/tts/__init__.py
@@ -504,7 +504,19 @@ def create():
         tts_module = config.get('tts', {}).get('module', 'mimic')
         tts_config = config.get('tts', {}).get(tts_module, {})
         tts_lang = tts_config.get('lang', lang)
-        clazz = TTSFactory.CLASSES.get(tts_module)
-        tts = clazz(tts_lang, tts_config)
-        tts.validator.validate()
+        try:
+            clazz = TTSFactory.CLASSES.get(tts_module)
+            tts = clazz(tts_lang, tts_config)
+            tts.validator.validate()
+        except Exception as e:
+            # Fallback to mimic if an error occurs while loading.
+            if tts_module != 'mimic':
+                LOG.exception('The selected TTS backend couldn\'t be loaded. '
+                              'Falling back to Mimic')
+                tts = Mimic(tts_lang, tts_config)
+                tts.validator.validate()
+            else:
+                LOG.exception('The TTS could not be loaded.')
+                raise
+
         return tts


Test patch:
Repository: MycroftAI/mycroft-core

Issue:
extractnumber_en fails with XX%
extractnumber_en returns `False` when a a number is followed by `%`, e.g. `100%`. The number by itself works correctly.

https://github.com/MycroftAI/mycroft-core/compare/bug/extract_number_percent provides a failing unit test.


Discussion:

Resolving patch:
diff --git a/mycroft/skills/mycroft_skill/mycroft_skill.py b/mycroft/skills/mycroft_skill/mycroft_skill.py
--- a/mycroft/skills/mycroft_skill/mycroft_skill.py
+++ b/mycroft/skills/mycroft_skill/mycroft_skill.py
@@ -675,9 +675,22 @@ def find_resource(self, res_name, res_dirname=None):
         Returns:
             string: The full path to the resource file or None if not found
         """
+        result = self._find_resource(res_name, self.lang, res_dirname)
+        if not result and self.lang != 'en-us':
+            # when resource not found try fallback to en-us
+            LOG.warning(
+                "Resource '{}' for lang '{}' not found: trying 'en-us'"
+                .format(res_name, self.lang)
+            )
+            result = self._find_resource(res_name, 'en-us', res_dirname)
+        return result
+
+    def _find_resource(self, res_name, lang, res_dirname=None):
+        """Finds a resource by name, lang and dir
+        """
         if res_dirname:
             # Try the old translated directory (dialog/vocab/regex)
-            path = join(self.root_dir, res_dirname, self.lang, res_name)
+            path = join(self.root_dir, res_dirname, lang, res_name)
             if exists(path):
                 return path
 
@@ -687,7 +700,7 @@ def find_resource(self, res_name, res_dirname=None):
                 return path
 
         # New scheme:  search for res_name under the 'locale' folder
-        root_path = join(self.root_dir, 'locale', self.lang)
+        root_path = join(self.root_dir, 'locale', lang)
         for path, _, files in walk(root_path):
             if res_name in files:
                 return join(path, res_name)


Test patch:
Repository: MycroftAI/mycroft-core

Issue:
`:log level debug` not working from CLI
On a Linux installation in the Mycroft CLI, issuing the `:log level debug` command changes the log level for output to `/var/log/mycroft/*.log` but has no effect on the messages shown in the CLI's log output. 

Tested on Ubuntu 19.10 and Debian 10



Discussion:
Can you verify that DEBUG messages aren't filtered? check the ~/.mycroft_cli.conf

Will have something like: `"filters": ["mouth.viseme", "mouth.display", "mouth.icon", "DEBUG"]`
I can confirm that they *are* filtered, and also that changing the log level to debug does not un-filter them.

Is that the intended behavior? I'd expect that filter to be removed (or bypassed) when the user selects debug logging.
tack så mycket, åke :-) 

```
kathyreid@kathyreid-zenbook-ux533fd:~$ cat .mycroft_cli.conf 
{"filters": ["mouth.viseme", "mouth.display", "mouth.icon", "DEBUG"], "cy_chat_area": 7, "show_last_key": false, "max_log_lines": 5000, "show_meter": true}
```
changing this now gives: 

![Screenshot from 2019-12-09 16-59-24](https://user-images.githubusercontent.com/114158/70411057-4521f200-1aa5-11ea-84a6-70ca3b97000e.png)

What I think is happening here is that the `log level debug` in the CLI _allows_ the logging to the shown in the CLI, but it's suppressed by `mycroft_cli.conf` - that is, the `.conf` file takes precedence.

It sort of is intended, the filter is separate system for the cli.

That said the DEBUG shouldn't be filtered by [default](https://github.com/MycroftAI/mycroft-core/blob/dev/mycroft/client/text/text_client.py#L65). so that's an issue. We could technically forbid the DEBUG to be filtered now since it can now be controlled via the log level.

Resolving patch:
diff --git a/mycroft/client/text/text_client.py b/mycroft/client/text/text_client.py
--- a/mycroft/client/text/text_client.py
+++ b/mycroft/client/text/text_client.py
@@ -62,7 +62,7 @@
 max_log_lines = 5000
 mergedLog = []
 filteredLog = []
-default_log_filters = ["mouth.viseme", "mouth.display", "mouth.icon", "DEBUG"]
+default_log_filters = ["mouth.viseme", "mouth.display", "mouth.icon"]
 log_filters = list(default_log_filters)
 log_files = []
 find_str = None
@@ -175,7 +175,8 @@ def load_settings():
         with io.open(config_file, 'r') as f:
             config = json.load(f)
         if "filters" in config:
-            log_filters = config["filters"]
+            # Disregard the filtering of DEBUG messages
+            log_filters = [f for f in config["filters"] if f != "DEBUG"]
         if "cy_chat_area" in config:
             cy_chat_area = config["cy_chat_area"]
         if "show_last_key" in config:


Test patch:
Repository: MycroftAI/mycroft-core

Issue:
Italian language with Google TTS
I'm running Mycroft on Manjaro Linux, with Italian language.
I tried both the "British male" and "American male" voices, and they do not speak Italian at all. So I decided to try the Google Voice.
That way, no sound is emitted. Whenever Mycroft tries to speak, I see this error in logs (file audio.log):

```
2020-04-13 10:45:39.632 | INFO     | 195922 | mycroft.audio.speech:mute_and_speak:127 | Speak: Va uno spettacolo
2020-04-13 10:45:40.070 | ERROR    | 195922 | mycroft.audio.speech:handle_speak:99 | Error in mute_and_speak
Traceback (most recent call last):
  File "/home/luke/git/mycroft-core/mycroft/audio/speech.py", line 95, in handle_speak
    mute_and_speak(chunk, ident, listen)
  File "/home/luke/git/mycroft-core/mycroft/audio/speech.py", line 129, in mute_and_speak
    tts.execute(utterance, ident, listen)
  File "/home/luke/git/mycroft-core/mycroft/tts/tts.py", line 337, in execute
    wav_file, phonemes = self.get_tts(sentence, wav_file)
  File "/home/luke/git/mycroft-core/mycroft/tts/google_tts.py", line 35, in get_tts
    tts = gTTS(text=sentence, lang=self.lang)
  File "/home/luke/git/mycroft-core/.venv/lib/python3.8/site-packages/gtts/tts.py", line 121, in __init__
    raise ValueError("Language not supported: %s" % lang)
ValueError: Language not supported: it-it
```

The problem is that Google TTS can handle "it", but it cannot handle "it-it".

I will release a PR shortly for fixing this. (So far, Google Voice seems my only option for Italian language, so I really need that).



Discussion:

Resolving patch:
diff --git a/mycroft/tts/google_tts.py b/mycroft/tts/google_tts.py
--- a/mycroft/tts/google_tts.py
+++ b/mycroft/tts/google_tts.py
@@ -13,13 +13,19 @@
 # limitations under the License.
 #
 from gtts import gTTS
+from gtts.lang import tts_langs
 
 from .tts import TTS, TTSValidator
 
+supported_langs = tts_langs()
+
 
 class GoogleTTS(TTS):
     """Interface to google TTS."""
     def __init__(self, lang, config):
+        if lang.lower() not in supported_langs and \
+                                     lang[:2].lower() in supported_langs:
+            lang = lang[:2]
         super(GoogleTTS, self).__init__(lang, config, GoogleTTSValidator(
             self), 'mp3')
 
@@ -42,8 +48,10 @@ def __init__(self, tts):
         super(GoogleTTSValidator, self).__init__(tts)
 
     def validate_lang(self):
-        # TODO
-        pass
+        lang = self.tts.lang
+        if lang.lower() not in supported_langs:
+            raise ValueError("Language not supported by gTTS: {}"
+                             .format(lang))
 
     def validate_connection(self):
         try:


Test patch:
Repository: MycroftAI/mycroft-core

Issue:
CommonIoTSkill subclasses can't initialize with register_entities_and_scenes
## To recreate:

Create a skill subclassing `CommonIoTSkill` and call register_entities_and_scenes` in the `initialize` method.

## Be as specific as possible about the expected condition, and the deviation from expected condition. 

This is the behavior described in the documentation, so we would expect it to register the entities and scenes provided. Instead, it errors because it attempts to `dig_for_message` when there is no originating message to find.

## Fix:

Revert the commit that used `dig_for_message` as it shouldn't attempt to do so in the `register_words` function.


Discussion:

Resolving patch:
diff --git a/mycroft/skills/common_iot_skill.py b/mycroft/skills/common_iot_skill.py
--- a/mycroft/skills/common_iot_skill.py
+++ b/mycroft/skills/common_iot_skill.py
@@ -456,11 +456,10 @@ def _register_words(self, words: [str], word_type: str):
             word_type:
         """
         if words:
-            message = dig_for_message()
-            self.bus.emit(message.forward(_BusKeys.REGISTER,
-                                          data={"skill_id": self.skill_id,
-                                                "type": word_type,
-                                                "words": list(words)}))
+            self.bus.emit(Message(_BusKeys.REGISTER,
+                                  data={"skill_id": self.skill_id,
+                                        "type": word_type,
+                                        "words": list(words)}))
 
     def register_entities_and_scenes(self):
         """


Test patch:
Repository: MycroftAI/mycroft-core

Issue:
skill keep reloading indefinitely if there are files with a future date
due to a temporary problem in my system, i ended up with a skill with __init.py modification time set in the future. this caused that skill to constantly be reloaded my mycroft core, and unloaded just after the loading was complted. took some hours of debug to understand this was actually the problem.

perhaps skills with files with modification date from the future should just be stopped from loading and have a debug log about it?


Discussion:
Maybe a max(read timestamp, current timestamp) could be used as a sanity filter?

Edit: No wait that won't work. Disregarding files with future timestamps completely maybe...

Resolving patch:
diff --git a/mycroft/skills/skill_loader.py b/mycroft/skills/skill_loader.py
--- a/mycroft/skills/skill_loader.py
+++ b/mycroft/skills/skill_loader.py
@@ -73,6 +73,19 @@ def load_skill_module(path, skill_id):
     return mod
 
 
+def _bad_mod_times(mod_times):
+    """Return all entries with modification time in the future.
+
+    Arguments:
+        mod_times (dict): dict mapping file paths to modification times.
+
+    Returns:
+        List of files with bad modification times.
+    """
+    current_time = time()
+    return [path for path in mod_times if mod_times[path] > current_time]
+
+
 def _get_last_modified_time(path):
     """Get the last modified date of the most recently updated file in a path.
 
@@ -99,6 +112,11 @@ def _get_last_modified_time(path):
                 all_files.append(os.path.join(root_dir, f))
 
     # check files of interest in the skill root directory
+    mod_times = {f: os.path.getmtime(f) for f in all_files}
+    # Ensure modification times are valid
+    bad_times = _bad_mod_times(mod_times)
+    if bad_times:
+        raise OSError('{} had bad modification times'.format(bad_times))
     if all_files:
         return max(os.path.getmtime(f) for f in all_files)
     else:
@@ -118,6 +136,8 @@ def __init__(self, bus, skill_directory):
         self.active = True
         self.config = Configuration.get()
 
+        self.modtime_error_log_written = False
+
     @property
     def is_blacklisted(self):
         """Boolean value representing whether or not a skill is blacklisted."""
@@ -135,10 +155,14 @@ def reload_needed(self):
         """
         try:
             self.last_modified = _get_last_modified_time(self.skill_directory)
-        except FileNotFoundError as e:
-            LOG.error('Failed to get last_modification time '
-                      '({})'.format(repr(e)))
+        except OSError as err:
             self.last_modified = self.last_loaded
+            if not self.modtime_error_log_written:
+                self.modtime_error_log_written = True
+                LOG.error('Failed to get last_modification time '
+                          '({})'.format(repr(err)))
+        else:
+            self.modtime_error_log_written = False
 
         modified = self.last_modified > self.last_loaded
 


Test patch:
Repository: MycroftAI/mycroft-core

Issue:
Idea: Enhance Amazon Polly support
Amazon Polly works well using standard voices, I have it running perfectly under the latest Picroft image. However, there is no current support for 'neural' engine voices, as well as 'conversational' style SSML. Both of these provide exceptionally high quality text-to-speech audio and would be nice to have the ability to use with Mycroft.

This [post](https://community.mycroft.ai/t/regarding-polly-tts-support/8722/10) on the community forums goes in to a little more detail on it.

Thanks!


Discussion:

Resolving patch:
diff --git a/mycroft/tts/polly_tts.py b/mycroft/tts/polly_tts.py
--- a/mycroft/tts/polly_tts.py
+++ b/mycroft/tts/polly_tts.py
@@ -34,6 +34,7 @@ def __init__(self, lang="en-us", config=None):
         self.key_id = self.config.get("access_key_id", '')
         self.key = self.config.get("secret_access_key", '')
         self.region = self.config.get("region", 'us-east-1')
+        self.engine = self.config.get("engine", "standard")
         self.polly = boto3.Session(aws_access_key_id=self.key_id,
                                    aws_secret_access_key=self.key,
                                    region_name=self.region).client('polly')
@@ -49,7 +50,8 @@ def get_tts(self, sentence, wav_file):
             OutputFormat=self.audio_ext,
             Text=sentence,
             TextType=text_type,
-            VoiceId=self.voice)
+            VoiceId=self.voice,
+            Engine=self.engine)
 
         with open(wav_file, 'wb') as f:
             f.write(response['AudioStream'].read())


Test patch: