Repository: explosion/spaCy

Issue:
Model download returns 0 on failure
While building a Docker image containing spaCy, there was an issue downloading the basic model with `python -m spacy download en`:

`ReadTimeoutError: HTTPSConnectionPool(host='github-production-release-asset-2e65be.s3.amazonaws.com', port=443): Read timed out.`

The download returned 0 so the build continued.  Next, while building an Rasa NLU model, it couldn't find the model (as you might expect):

`IOError: Can't find model 'en'`

So I thought, if the download doesn't report failure, maybe I should use the `validate` command to make sure there's a model present before continuing the build?  But `validate` also returns 0 when no models are present:

```
    Installed models (spaCy v2.0.5)
    /usr/local/lib/python2.7/dist-packages/spacy


    No models found in your current environment.
```

I would expect the download tool to return non-zero when it fails to finish downloading a model.

## Your Environment
<!-- Include details of your environment. If you're using spaCy 1.7+, you can also type 
`python -m spacy info --markdown` and copy-paste the result here.-->
* **Python version:** 2.7.12
* **Platform:** Linux-4.4.0-98-generic-x86_64-with-Ubuntu-16.04-xenial
* **spaCy version:** 2.0.5
* Environment Information: building an image with Docker 17.09.0



Discussion:
Thanks – good point, I didn't even realise the `download` command currently behaves like this – this should definitely be fixed. The `validate` command was originally intended as more of a user-facing utility that prints nicely formatted and helpful info about the models, so we didn't really consider the automated usage and exit codes here... but we might as well do it properly, so thanks for the suggestion!

Btw, if you're downloading models as part of an automated process, you can also just run `pip install` directly, and use the URL of the model archive (see the [model releases](https://spacy.io/usage/models#production)). This lets you download the exact model and model version you need, and saves you the extra roundtrip to the spaCy compatibility table. Instead of calling `spacy.load()`, you can also import the model as a module:

```python
import en_core_web_sm
nlp = en_core_web_sm.load()
```

In some cases, it might be a little nicer to get a more "native" `ImportError` if the model isn't installed, rather than a spaCy error somewhere down the line. (But this also depends on your personal preference. You can find more details on this in the ["Using models in production"](https://spacy.io/usage/models#production) section in the docs.)

Resolving patch:
diff --git a/spacy/cli/download.py b/spacy/cli/download.py
--- a/spacy/cli/download.py
+++ b/spacy/cli/download.py
@@ -31,24 +31,28 @@ def download(model, direct=False):
         version = get_version(model_name, compatibility)
         dl = download_model('{m}-{v}/{m}-{v}.tar.gz'.format(m=model_name,
                                                             v=version))
-        if dl == 0:
-            try:
-                # Get package path here because link uses
-                # pip.get_installed_distributions() to check if model is a
-                # package, which fails if model was just installed via
-                # subprocess
-                package_path = get_package_path(model_name)
-                link(model_name, model, force=True, model_path=package_path)
-            except:
-                # Dirty, but since spacy.download and the auto-linking is
-                # mostly a convenience wrapper, it's best to show a success
-                # message and loading instructions, even if linking fails.
-                prints(
-                    "Creating a shortcut link for 'en' didn't work (maybe "
-                    "you don't have admin permissions?), but you can still "
-                    "load the model via its full package name:",
-                    "nlp = spacy.load('%s')" % model_name,
-                    title="Download successful")
+        if dl != 0:
+            # if download subprocess doesn't return 0, exit with the respective
+            # exit code before doing anything else
+            sys.exit(dl)
+        try:
+            # Get package path here because link uses
+            # pip.get_installed_distributions() to check if model is a
+            # package, which fails if model was just installed via
+            # subprocess
+            package_path = get_package_path(model_name)
+            link(None, model_name, model, force=True,
+                    model_path=package_path)
+        except:
+            # Dirty, but since spacy.download and the auto-linking is
+            # mostly a convenience wrapper, it's best to show a success
+            # message and loading instructions, even if linking fails.
+            prints(
+                "Creating a shortcut link for 'en' didn't work (maybe "
+                "you don't have admin permissions?), but you can still "
+                "load the model via its full package name:",
+                "nlp = spacy.load('%s')" % model_name,
+                title="Download successful but linking failed")
 
 
 def get_json(url, desc):
@@ -84,5 +88,5 @@ def get_version(model, comp):
 def download_model(filename):
     download_url = about.__download_url__ + '/' + filename
     return subprocess.call(
-        [sys.executable, '-m', 'pip', 'install', '--no-cache-dir',
+        [sys.executable, '-m', 'pip', 'install', '--no-cache-dir', '--no-deps',
          download_url], env=os.environ.copy())
diff --git a/spacy/cli/link.py b/spacy/cli/link.py
--- a/spacy/cli/link.py
+++ b/spacy/cli/link.py
@@ -34,11 +34,18 @@ def link(origin, link_name, force=False, model_path=None):
                "located here:", path2str(spacy_loc), exits=1,
                title="Can't find the spaCy data path to create model symlink")
     link_path = util.get_data_path() / link_name
-    if link_path.exists() and not force:
+    if link_path.is_symlink() and not force:
         prints("To overwrite an existing link, use the --force flag.",
                title="Link %s already exists" % link_name, exits=1)
-    elif link_path.exists():
+    elif link_path.is_symlink():  # does a symlink exist?
+        # NB: It's important to check for is_symlink here and not for exists,
+        # because invalid/outdated symlinks would return False otherwise.
         link_path.unlink()
+    elif link_path.exists(): # does it exist otherwise?
+        # NB: Check this last because valid symlinks also "exist".
+        prints("This can happen if your data directory contains a directory "
+               "or file of the same name.", link_path,
+               title="Can't overwrite symlink %s" % link_name, exits=1)
     try:
         symlink_to(link_path, model_path)
     except:
diff --git a/spacy/cli/validate.py b/spacy/cli/validate.py
--- a/spacy/cli/validate.py
+++ b/spacy/cli/validate.py
@@ -4,6 +4,7 @@
 import requests
 import pkg_resources
 from pathlib import Path
+import sys
 
 from ..compat import path2str, locale_escape
 from ..util import prints, get_data_path, read_json
@@ -62,6 +63,9 @@ def validate():
                "them from the data directory. Data path: {}"
                .format(path2str(get_data_path())))
 
+    if incompat_models or incompat_links:
+        sys.exit(1)
+
 
 def get_model_links(compat):
     links = {}


Test patch: