Repository: MycroftAI/mycroft-core

Issue:
Initial load of skills fails if auto_update is true.
On a new clone of the dev tree on an Ubuntu 16.04 machine **that's never run mycroft before.**

I ran _msm/msm default_ and it downloaded 27 skills, but mycroft wouldn't load any skills except _pairing_. I looked at the skills loading code and saw it wasn't getting past this:

```
    if (exists(SKILLS_DIR) and
          (self.next_download or not update)):
```

After I changed my config to _auto_update: false_, they loaded.






Discussion:
Hi tlc, thanks for reporting. Just to make certain, the machine is connected to the internet?

The skills should load as soon as an skill update is scheduled. This will occur as soon as the device detects an internet connection. If the device isn't intended to be connected the auto-update needs to be turned off.

In any case some sort of timeout for the signal might be an idea to make this more reliable... This is mainly necessary for the Mark-1 device since there's some issues with time syncing occuring while skills are loading and may be simplified for the desktop.
It is on the internet.  Mycroft is working (accessing the mycroft proxy for SST, etc.) and I'm posting from the machine now.

But I am seeing:

> HTTPSConnectionPool(host='api.mycroft.ai', port=443): Read timed out. (read timeout=15)
> 13:17:34.810 - mycroft.skills.settings:_poll_skill_settings:401 - ERROR - Failed to fetch skill settings:
> HTTPSConnectionPool(host='api.mycroft.ai', port=443): Read timed out. (read timeout=15)
> 13:17:50.501 - mycroft.skills.settings:_poll_skill_settings:401 - ERROR - Failed to fetch skill settings:
> HTTPSConnectionPool(host='api.mycroft.ai', port=443): Read timed out. (read timeout=15)

It's worth noting that I'm behind a proxy.  But again, mycroft is working.  It just streamed me NPR.  So unless the skills portion doesn't use the same access code...

Also, I didn't see the above messages *before* I changed the setting and got the skills loaded.  I only saw this:

> Log Output: 0-18 of 18
> ================================================================================================================================= mycroft-core 18.2.0 ===
> 09:53:14.925 - mycroft.messagebus.client.ws:on_open:58 - INFO - Connected
> 09:53:14.937 - mycroft.skills.core:load_skill:106 - INFO - ATTEMPTING TO LOAD SKILL: skill-pairing with ID -5844316681041254434
> 09:53:14.944 - mycroft.skills.core:load_skill:124 - INFO - Loaded skill-pairing
> ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
> ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
> ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
> ALSA lib pcm_route.c:867:(find_matching_chmap) Found no matching channel map
> 09:53:16.129 - mycroft.client.speech.listener:create_wake_word_recognizer:235 - INFO - creating wake word engine
> 09:53:16.131 - mycroft.client.speech.hotword_factory:create_hotword:268 - INFO - creating hey mycroft
> 09:53:16.168 - mycroft.client.speech.listener:create_wakeup_recognizer:253 - INFO - creating stand up word engine
> 09:53:16.169 - mycroft.client.speech.hotword_factory:create_hotword:268 - INFO - creating wake up
> 09:53:16.870 - mycroft.messagebus.client.ws:on_open:58 - INFO - Connected
> 09:53:38.277 - main:handle_record_begin:36 - INFO - Begin Recording…
> 09:53:39.538 - main:handle_record_end:41 - INFO - End Recording…
> 09:53:39.546 - mycroft.session:get:76 - INFO - New Session Start: 3d4a0f50-844d-4bec-9516-b210b742b1d7
> 09:53:39.546 - main:handle_wakeword:57 - INFO - Wakeword Detected: hey mycroft
> 09:53:40.662 - main:handle_utterance:62 - INFO - Utterance: [u’tell me a joke’]
> ^— NEWEST —^


I see the skills code handling msm events.  I'm not sure, but it's possible I ran _msm/msm default_ after stopping mycroft.   

There's a lack of info on running msm.  Was I even supposed to or does Mycroft **down**load the defaults by itself?  I only found it in the forum.  It's not mentioned in the README here or the [Mycroft for Linux](https://mycroft.ai/documentation/linux/) page.
The skills process installs and updates the skills at startup (if it wasn't done recently), takes upto a minute or so in the worst case. There is no harm in running msm manually (as long as mycroft isn't is running it at the same time)

Basically it's locked from loading the skills (except priority skills like pairing) until msm has finished checking for updates.

https://github.com/MycroftAI/mycroft-core/blob/dev/mycroft/util/__init__.py#L133 is used to determine if it's connected or not.

Is this a constant thing or does it work some times? If it's constant can you try to run the code linked above and see if there's an issue there?
Is what a constant thing?

(I said True earlier, but I was in the wrong window.)

> $ python /tmp/conn.py
> False
I don't believe this code is proxy friendly.

My company has proxies for http/s.  We use our own DNS.  Outside DNS is blocked.

> $ nslookup google.com 8.8.8.8
> ;; connection timed out; no servers could be reached



Thanks @tlc good info. Will look into alternatives. Personally I always check sunet.se when checking internet connectivity.

I suppose the target for msm (currently github) could be checked so `connected() or github_available()` might be a way forward...
Just to be clear, it's not the host it's the port.  Port 53 is blocked.   IMHO, you need to try an HTTP request somewhere.

Resolving patch:
diff --git a/mycroft/util/__init__.py b/mycroft/util/__init__.py
--- a/mycroft/util/__init__.py
+++ b/mycroft/util/__init__.py
@@ -22,6 +22,7 @@
 import os.path
 import psutil
 from stat import S_ISREG, ST_MTIME, ST_MODE, ST_SIZE
+import requests
 
 import signal as sig
 
@@ -136,18 +137,41 @@ def read_dict(filename, div='='):
     return d
 
 
-def connected(host="8.8.8.8", port=53, timeout=3):
+def connected():
+    """ Check connection by connecting to 8.8.8.8, if this is
+    blocked/fails, Microsoft NCSI is used as a backup
+
+    Returns:
+        True if internet connection can be detected
+    """
+    return connected_dns() or connected_ncsi()
+
+
+def connected_ncsi():
+    """ Check internet connection by retrieving the Microsoft NCSI endpoint.
+
+    Returns:
+        True if internet connection can be detected
     """
-    Thanks to 7h3rAm on
-    Host: 8.8.8.8 (google-public-dns-a.google.com)
-    OpenPort: 53/tcp
-    Service: domain (DNS/TCP)
-
-    NOTE:
-    This is no longer in use by this version
-    New method checks for a connection using ConnectionError only when
-    a question is asked
+    try:
+        r = requests.get('http://www.msftncsi.com/ncsi.txt')
+        if r.text == u'Microsoft NCSI':
+            return True
+    except Exception:
+        pass
+    return False
+
+
+def connected_dns(host="8.8.8.8", port=53, timeout=3):
+    """ Check internet connection by connecting to DNS servers
+
+    Returns:
+        True if internet connection can be detected
     """
+    # Thanks to 7h3rAm on
+    # Host: 8.8.8.8 (google-public-dns-a.google.com)
+    # OpenPort: 53/tcp
+    # Service: domain (DNS/TCP)
     try:
         socket.setdefaulttimeout(timeout)
         socket.socket(socket.AF_INET, socket.SOCK_STREAM).connect((host, port))


Test patch: