Repository: MycroftAI/mycroft-core

Issue:
Skill Shutdown sometimes fails causing duplicate instances of Skill
## Devices:
Issue occurred on two separate Debian Linux installations
- one running Mycroft-core 19.02.14, 
- the other running dev branch the date of this issue.

## Steps to reproduce:
1. Load Mycroft and ensure working as expected
2. Make intentional code error in Alarm Skill to raise exception eg `self.log.info(not_a_real_var)`
3. Skill fails on reload with the following error:
```
 12:04:25.758 | INFO     | 22386 | mycroft.skills.skill_loader:reload:109 | ATTEMPTING TO RELOAD SKILL: mycroft-alarm.mycroftai
 12:04:25.760 | ERROR    | 22386 | mycroft.skills.skill_loader:_execute_instance_shutdown:145 | An error occurred while shutting down AlarmSkill
Traceback (most recent call last):
  File "/home/fs-neriahbjato/Documents/Full_Scale/Mycroft/mycroft-core/mycroft/skills/skill_loader.py", line 142, in _execute_instance_shutdown
    self.instance.default_shutdown()
  File "/home/fs-neriahbjato/Documents/Full_Scale/Mycroft/mycroft-core/mycroft/skills/mycroft_skill/mycroft_skill.py", line 1162, in default_shutdown
    self.event_scheduler.shutdown()
  File "/home/fs-neriahbjato/Documents/Full_Scale/Mycroft/mycroft-core/mycroft/skills/event_scheduler.py", line 433, in shutdown
    self.events.clear()
  File "/home/fs-neriahbjato/Documents/Full_Scale/Mycroft/mycroft-core/mycroft/skills/mycroft_skill/event_container.py", line 183, in clear
    self.bus.remove(e, f)
  File "/home/fs-neriahbjato/Documents/Full_Scale/Mycroft/mycroft-core/mycroft/messagebus/client/client.py", line 172, in remove
    self.emitter.remove_listener(event_name, func)
  File "/home/fs-neriahbjato/Documents/Full_Scale/Mycroft/mycroft-core/mycroft/messagebus/client/threaded_event_emitter.py", line 57, in remove_listener
    return super().remove_listener(event_name, func)
  File "/home/fs-neriahbjato/Documents/Full_Scale/Mycroft/mycroft-core/.venv/lib/python3.6/site-packages/pyee/__init__.py", line 205, in remove_listener
    self._events[event].pop(f)
KeyError: <function create_basic_wrapper.<locals>.wrapper at 0x7f2a504781e0>
```
4. Remove bad code and save to reload Skill
```
 12:04:25.803 | INFO     | 22386 | mycroft.skills.skill_loader:_communicate_load_status:273 | Skill mycroft-alarm.mycroftai loaded successfully
```
5. Trigger the Skill with an utterance eg "what alarms are set"
6. Mycroft responds twice
  - once from the failed instance of the Skill: "An error occurred while processing a request in Alarm Skill"
  - once from the new working instance of the Skill: "No alarms have been set"

The failed instance of the Skill will remain until Mycroft services are restarted.
After restarting, the same steps cause the same error.

## Investigation:
This was only occurring for the Alarm Skill, did not get the same behavior in the Date Time Skill.
To see what might be causing it, I added some logging to the offending function in [`mycroft/messagebus/client/threaded_event_emitter.py`](https://github.com/MycroftAI/mycroft-core/blob/dev/mycroft/messagebus/client/threaded_event_emitter.py):
```
def remove_listener(self, event_name, func):
        """ Wrap the remove to translate from function to wrapped
            function.
        """
        for w in self.wrappers[event_name]:
            if w[0] == func:
                self.wrappers[event_name].remove(w)
                return super().remove_listener(event_name, w[1])
        # if no wrapper exists try removing the function
        LOG.info("REMOVING LISTENER")
        LOG.info("event_name: " + str(event_name))
        LOG.info(event_name)
        LOG.info("func: " + str(func))
        LOG.info(func)
        return super().remove_listener(event_name, func)
```
Then reloaded Mycroft and can no longer trigger the issue.

## Hypothesis
- Presumably the `func` was not being found in `self.wrappers` allowing it to proceed to try removing the function itself on [line 57](https://github.com/MycroftAI/mycroft-core/blob/dev/mycroft/messagebus/client/threaded_event_emitter.py#L57) which then failed.
- Possibly the function was modified or removed from `self.wrappers` causing it to not match?
- Is `self.wrappers` cached between service restarts, and would editing this file have recreated it?


As I'm no longer able to reproduce the error it's difficult to continue debugging. So documenting the situation here in case anyone else runs into it and we can pick it back up.

Unless anyone else has an idea of what might be happening?


Discussion:
Just to make sure I understand correctly, this is when a skill fails to load right? or has it been observed after normal shutdown? I guessthe injected error should be in the initialize method?

I managed to trigger it as well, and it might be related to an issue I'm tracking where the same event is removed twice when it's shutting down...
Did a bit of tracing and it's not the same issue. Could possibly be a race condition for a scheduled *once* event...

Resolving patch:
diff --git a/mycroft/skills/mycroft_skill/event_container.py b/mycroft/skills/mycroft_skill/event_container.py
--- a/mycroft/skills/mycroft_skill/event_container.py
+++ b/mycroft/skills/mycroft_skill/event_container.py
@@ -138,9 +138,12 @@ def once_wrapper(message):
         if handler:
             if once:
                 self.bus.once(name, once_wrapper)
+                self.events.append((name, once_wrapper))
             else:
                 self.bus.on(name, handler)
-            self.events.append((name, handler))
+                self.events.append((name, handler))
+
+            LOG.debug('Added event: {}'.format(name))
 
     def remove(self, name):
         """Removes an event from bus emitter and events list.


Test patch:
diff --git a/test/unittests/skills/test_event_container.py b/test/unittests/skills/test_event_container.py
--- a/test/unittests/skills/test_event_container.py
+++ b/test/unittests/skills/test_event_container.py
@@ -32,12 +32,13 @@ def test_add(self):
         self.assertTrue(bus.on.called)
 
         # Test add single shot event handler
+        len_before = len(container.events)
         container.add('test2', example_handler, once=True)
+        self.assertEqual(len_before + 1, len(container.events))
         self.assertTrue(bus.once.called)
 
         # Verify correct content in event container
         self.assertTrue(('test1', example_handler) in container.events)
-        self.assertTrue(('test2', example_handler) in container.events)
         self.assertEqual(len(container.events), 2)
 
     def test_remove(self):