Repository: spack/spack

Issue:
Updated spack install tries to get tarballs off internet first before looking in local mirror!
CC: @fryeguy52 

Tag: snl-atdm-issue

The older version of spack we were using would try to get source off our local mirror before going out to the internet with behavior like:

```
==> Installing libsigsegv
==> Searching for binary cache of libsigsegv
==> Finding buildcaches in /ascldap/users/rabartl/Spack.base/atdm-spack/atdm-spack-tarball-mirror/build_cache
==> No binary for libsigsegv found: installing from source
==> Fetching file:///ascldap/users/rabartl/Spack.base/atdm-spack/atdm-spack-tarball-mirror/libsigsegv/libsigsegv-2.11.tar.gz
==> Staging archive: /home/rabartl/Spack.base/atdm-spack/spack/var/spack/stage/libsigsegv-2.11-xqiyghtlms7lwgexu3wh7zdtbw6lnytl/libsigsegv-2.11.tar.gz
==> Created stage in /home/rabartl/Spack.base/atdm-spack/spack/var/spack/stage/libsigsegv-2.11-xqiyghtlms7lwgexu3wh7zdtbw6lnytl
==> No patches needed for libsigsegv
==> Building libsigsegv [AutotoolsPackage]
==> Executing phase: 'autoreconf'
==> Executing phase: 'configure'
==> Executing phase: 'build'
==> Executing phase: 'install'
==> Successfully installed libsigsegv
  Fetch: 0.02s.  Build: 4.54s.  Total: 4.56s.
[+] /home/rabartl/Spack.base/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/libsigsegv-2.11-xqiyghtlms7lwgexu3wh7zdtbw6lnytl
```

But with the upgrade version of spack 0.13.0, it does:

```
==> Installing libsigsegv
==> Searching for binary cache of libsigsegv
==> Finding buildcaches in /ascldap/users/rabartl/Spack.base4/atdm-spack/atdm-spack-tarball-mirror/build_cache
==> No binary for libsigsegv found: installing from source
curl: (7) Failed to connect to 2001:470:142:5::200: Network is unreachable
==> Fetching https://ftpmirror.gnu.org/libsigsegv/libsigsegv-2.12.tar.gz
==> Fetching from https://ftpmirror.gnu.org/libsigsegv/libsigsegv-2.12.tar.gz failed.
==> Fetching file:///ascldap/users/rabartl/Spack.base4/atdm-spack/atdm-spack-tarball-mirror/_source-cache/archive/3a/3ae1af359eebaa4ffc5896a1aee3568c052c99879316a1ab57f8fe1789c390b6.tar.gz
==> Staging archive: /tmp/rabartl/spack-stage/spack-stage-libsigsegv-2.12-fx22pfx4m4zrul7ahkzfhbvgud3t5j4q/libsigsegv-2.12.tar.gz
==> Created stage in /tmp/rabartl/spack-stage/spack-stage-libsigsegv-2.12-fx22pfx4m4zrul7ahkzfhbvgud3t5j4q
==> No patches needed for libsigsegv
==> Building libsigsegv [AutotoolsPackage]
==> Executing phase: 'autoreconf'
==> Executing phase: 'configure'
==> Executing phase: 'build'
==> Executing phase: 'install'
==> Successfully installed libsigsegv
  Fetch: 1m 3.33s.  Build: 4.13s.  Total: 1m 7.46s.
[+] /home/rabartl/Spack.base4/atdm-spack/spack/opt/spack/linux-rhel6-x86_64/gcc-4.4.7/libsigsegv-2.12-fx22pfx4m4zrul7ahkzfhbvgud3t5j4q
```

See that, over 1m to try to get the source from the internet (and failing because if did not set the proxies on purpose) and 4s to build!  This is not good.  This is killing out build times.

Is this a defect in Spack or has this behavior changed in spack for some good reason?  Is there a way to get spack install to not try to download stuff from the internet?






Discussion:
 I see the same behavior moving from v0.12.1 to v0.13.0.  
> I see the same behavior moving from v0.12.1 to v0.13.0.

@cwsmith, thanks for the confirmation!

Looks like spack may be missing some automated tests for dealing with mirrors (or this behavior was changed on purpose for some reason).
@bartlettroscoe Is it possibly a mirror precedence issue?  Can you share your list of configured mirrors?  I don't know if `spack mirror list` will list them in the order they'll be searched, but I think showing the contents of the `mirrors.yaml` should indicate that order.
This was changed in https://github.com/spack/spack/pull/11117. We anticipated switching it back eventually but did not anticipate that it would cause problems for users.

So @opadron I think we should prioritize this higher and add it to 0.13.1. You mentioned in https://github.com/spack/spack/pull/11117/files#r272646885 that you rearranged it because you were observing failures with the previous order. Do you happen to have the error messages (or would you be able to reproduce them quickly)?

I think in addition to restoring the old order, we should update our `curl` invocations to time out faster if we can't connect: 5 seconds seems like a reasonable amount of time to wait.
@scottwittenburg I got here a little late: if you look at https://github.com/spack/spack/pull/11117/files#r272646885 you'll see we in fact did change this ordering in Spack.
I am okay with spack changing the ordering by default but there needs to be a way to tell spack to not to try to download things off the internet (or tell it to prefer mirrors first).

Can we just add an option to spack install (or setting it in spack.yaml) to not try to download anything off the internet?  There are some organizations that can't allow any internet access and we need to be able to cleanly tell spack to not do so.

> Can we just add an option to spack install (or setting it in spack.yaml) to not try to download anything off the internet?

We could, although that would be different from the behavior before #11117 too: before that PR, if the mirror URLs failed, it would eventually try the default internet download URL.

There appears to be some logic present to support this but it's incomplete since there is no config or command line option available to trigger that behavior.

Resolving patch:
diff --git a/lib/spack/spack/fetch_strategy.py b/lib/spack/spack/fetch_strategy.py
--- a/lib/spack/spack/fetch_strategy.py
+++ b/lib/spack/spack/fetch_strategy.py
@@ -1129,7 +1129,7 @@ def fetch(self):
 
         parsed_url = url_util.parse(self.url)
         if parsed_url.scheme != 's3':
-            raise ValueError(
+            raise FetchError(
                 'S3FetchStrategy can only fetch from s3:// urls.')
 
         tty.msg("Fetching %s" % self.url)
@@ -1395,7 +1395,7 @@ class NoCacheError(FetchError):
 
 
 class FailedDownloadError(FetchError):
-    """Raised wen a download fails."""
+    """Raised when a download fails."""
     def __init__(self, url, msg=""):
         super(FailedDownloadError, self).__init__(
             "Failed to fetch file from URL: %s" % url, msg)
diff --git a/lib/spack/spack/stage.py b/lib/spack/spack/stage.py
--- a/lib/spack/spack/stage.py
+++ b/lib/spack/spack/stage.py
@@ -433,11 +433,9 @@ def fetch(self, mirror_only=False):
 
             # Add URL strategies for all the mirrors with the digest
             for url in urls:
-                fetchers.append(fs.from_url_scheme(
-                    url, digest, expand=expand, extension=extension))
-                # fetchers.insert(
-                #     0, fs.URLFetchStrategy(
-                #         url, digest, expand=expand, extension=extension))
+                fetchers.insert(
+                    0, fs.from_url_scheme(
+                        url, digest, expand=expand, extension=extension))
 
             if self.default_fetcher.cachable:
                 for rel_path in reversed(list(self.mirror_paths)):
diff --git a/lib/spack/spack/util/web.py b/lib/spack/spack/util/web.py
--- a/lib/spack/spack/util/web.py
+++ b/lib/spack/spack/util/web.py
@@ -177,7 +177,12 @@ def read_from_url(url, accept_content_type=None):
 
     # Do the real GET request when we know it's just HTML.
     req.get_method = lambda: "GET"
-    response = _urlopen(req, timeout=_timeout, context=context)
+
+    try:
+        response = _urlopen(req, timeout=_timeout, context=context)
+    except URLError as err:
+        raise SpackWebError('Download failed: {ERROR}'.format(
+            ERROR=str(err)))
 
     if accept_content_type and not is_web_url:
         content_type = response.headers.get('Content-type')


Test patch: