Repository: spack/spack

Issue:
spack assumes partial downloads are complete
If a file of the correct name exists, spack assumes the download is complete.  As a result, interrupted downloads cause subsequent install requests to fail rather than attempt to resume or retry the download:

```
rzalastor1@scogland:spack install -v llvm@3.7.0
==> Installing llvm
==> python is already installed in /g/g12/scogland/spack/opt/spack/chaos_5_x86_64_ib/gcc-4.9.2/python-2.7.11-tnnb5474rktee6vfl5ffmefvlbqgrqfa.
==> ncurses is already installed in /g/g12/scogland/spack/opt/spack/chaos_5_x86_64_ib/gcc-4.9.2/ncurses-6.0-2v7r63atwq6aw3p66bc3mkp7hxeoxgqx.
==> Installing swig
==> pcre is already installed in /g/g12/scogland/spack/opt/spack/chaos_5_x86_64_ib/gcc-4.9.2/pcre-8.38-qehu2lt63bbtou3oxh2eflua2o4u5osz.
==> Trying to fetch from http://prdownloads.sourceforge.net/swig/swig-3.0.2.tar.gz
#                                                                          1.6%^C
==> Error: Keyboard interrupt.
rzalastor1@scogland:spack install -v llvm@3.7.0
==> Installing llvm
==> python is already installed in /g/g12/scogland/spack/opt/spack/chaos_5_x86_64_ib/gcc-4.9.2/python-2.7.11-tnnb5474rktee6vfl5ffmefvlbqgrqfa.
==> ncurses is already installed in /g/g12/scogland/spack/opt/spack/chaos_5_x86_64_ib/gcc-4.9.2/ncurses-6.0-2v7r63atwq6aw3p66bc3mkp7hxeoxgqx.
==> Installing swig
==> pcre is already installed in /g/g12/scogland/spack/opt/spack/chaos_5_x86_64_ib/gcc-4.9.2/pcre-8.38-qehu2lt63bbtou3oxh2eflua2o4u5osz.
==> Already downloaded /g/g12/scogland/spack/var/spack/stage/swig-3.0.2-i67shydfa46bm24nz5nb2sfnitaarhor/swig-3.0.2.tar.gz.
==> Error: md5 checksum failed for /g/g12/scogland/spack/var/spack/stage/swig-3.0.2-i67shydfa46bm24nz5nb2sfnitaarhor/swig-3.0.2.tar.gz.
Expected 62f9b0d010cef36a13a010dc530d0d41 but got 92bba45c02b616b68900af861c0ee0e0.
```



Discussion:
As a note here, I'm not doing it right now, but this should be fixable by just swapping `-O` for `'-C', '-o', spack.archive_file + '.part'` in the curl args and moving the file to the final location on success.

Since downloads that fail the checksum tests will never be used, wouldn't it make sense to delete them automatically?

It currently does delete them automatically if it is allowed to clean up, but if not they stick around, fail the check, and must be manually cleaned.  The `-C` actually allows a partial download to resume rather than starting over.  If that's not sufficiently useful then a one-time remove and retry might be the way to go.

I like the partial download thing.  That seems very useful.  Is that supported in most curl versions?

As for deleting the tarball, I guess the reason I don't want to delete it is that in my experience with macports, checksums have failed simply because someone didn't update them.  If you delete the archive, re-downloading and getting the checksum is a pain.  Could move the failed archive to something like ".failed", I suppose, but isn't it simple enough to call `spack clean` if the checksum fails?  

As far as I know yes.  It's supported on 7.19.7, which happens to be the one we default to in LC, from 2009.

It does seem reasonable, but I'm thinking that having the download in a partial state would let it attempt to resume, then move, then checksum so there's a state between "started download" and "download assumed finished".  Once it's been moved, if the checksum failed then something has pretty clearly gone wrong.

@trws: I like your suggestion.  @eschnett: if we do Tom's thing, then Spack prints a message about the archive, with a path and maybe a suggestion to run `spack clean`, would that be enough?

Spack already has a check for the most common checksum failure case -- when a web gateway gives you HTML instead of a tarball.  It actually inspects the file and tries to see if it looks like HTML.  If we handle this and the case where the tarball just didn't download, then I suspect that if the checksum fails, human intervention is usually required and having the offending tarball is probably useful. so I'm hesitant to remove it.  Am I missing any cases?

@tgamblin I'm happy any way if manually aborted downloads are handled better.


Resolving patch:
diff --git a/lib/spack/spack/fetch_strategy.py b/lib/spack/spack/fetch_strategy.py
--- a/lib/spack/spack/fetch_strategy.py
+++ b/lib/spack/spack/fetch_strategy.py
@@ -157,12 +157,26 @@ def fetch(self):
             tty.msg("Already downloaded %s" % self.archive_file)
             return
 
+        possible_files = self.stage.expected_archive_files
+        save_file = None
+        partial_file = None
+        if possible_files:
+            save_file = self.stage.expected_archive_files[0]
+            partial_file = self.stage.expected_archive_files[0] + '.part'
+
         tty.msg("Trying to fetch from %s" % self.url)
 
-        curl_args = ['-O',  # save file to disk
+        if partial_file:
+            save_args = ['-C', '-',          # continue partial downloads
+                         '-o', partial_file] # use a .part file
+        else:
+            save_args = ['-O']
+
+        curl_args = save_args + [
                      '-f',  # fail on >400 errors
                      '-D', '-',  # print out HTML headers
-                     '-L', self.url, ]
+                     '-L',  # resolve 3xx redirects
+                     self.url, ]
 
         if sys.stdout.isatty():
             curl_args.append('-#')  # status bar when using a tty
@@ -178,6 +192,9 @@ def fetch(self):
             if self.archive_file:
                 os.remove(self.archive_file)
 
+            if partial_file and os.path.exists(partial_file):
+                os.remove(partial_file)
+
             if spack.curl.returncode == 22:
                 # This is a 404.  Curl will print the error.
                 raise FailedDownloadError(
@@ -209,6 +226,9 @@ def fetch(self):
                      "'spack clean <package>' to remove the bad archive, then fix",
                      "your internet gateway issue and install again.")
 
+        if save_file:
+            os.rename(partial_file, save_file)
+
         if not self.archive_file:
             raise FailedDownloadError(self.url)
 
diff --git a/lib/spack/spack/stage.py b/lib/spack/spack/stage.py
--- a/lib/spack/spack/stage.py
+++ b/lib/spack/spack/stage.py
@@ -210,6 +210,18 @@ def _need_to_create_path(self):
 
         return False
 
+    @property
+    def expected_archive_files(self):
+        """Possible archive file paths."""
+        paths = []
+        if isinstance(self.fetcher, fs.URLFetchStrategy):
+            paths.append(os.path.join(self.path, os.path.basename(self.fetcher.url)))
+
+        if self.mirror_path:
+            paths.append(os.path.join(self.path, os.path.basename(self.mirror_path)))
+
+        return paths
+
     @property
     def archive_file(self):
         """Path to the source archive within this stage directory."""


Test patch: