Repository: spack/spack

Issue:
Unit tests failing on Travis with Python 3.7
Unit tests are failing on Travis for Python 3.7. 

### Steps to reproduce the issue
Any PR or build of the `develop` branch since October 21st started failing for Python 3.7. The error could be reproduced locally by just using python 3.7 as an interpreter:
```console
$ python3.7 
Python 3.7.1 (default, Oct 21 2018, 04:01:51) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

$ python3.7 bin/spack test
...
web.py:133: AssertionError
_____________________________________________________ test_find_exotic_versions_of_archive_3 _____________________________________________________

    def test_find_exotic_versions_of_archive_3():
        versions = find_versions_of_archive(root_tarball, root, list_depth=3)
>       assert ver('2.0.0b2') in versions
E       AssertionError: assert Version('2.0.0b2') in {Version('0.0.0'): 'file:///home/mculpo/PycharmProjects/spack/lib/spack/spack/test/data/web/foo-0.0.0.tar.gz'}
E        +  where Version('2.0.0b2') = ver('2.0.0b2')

web.py:141: AssertionError
========================================= 9 failed, 1182 passed, 83 skipped, 1 xfailed in 221.95 seconds =========================================
```


The [last successful build](https://travis-ci.org/spack/spack/builds/444198345) appears to be have the same hash as the [first failing build](https://travis-ci.org/spack/spack/builds/444484951). 

### Error Message
There are 9 tests that seems to be failing. The log can be found [here](https://travis-ci.org/spack/spack/jobs/444484960). 


### Information on your system

The errors have been reproduced on an Ubuntu 18.04 machine and are persistent in Travis.


Discussion:
@baberlevi 
Apparently Python 3.7 raises an exception at this line:
https://github.com/spack/spack/blob/4ba3c81bc84a7cacbd3a8d159b22eacc031baca9/lib/spack/spack/util/web.py#L170

Investigating more deeply, it seems that the culprit is https://github.com/python/cpython/commit/97bfe8d3ebb0a54c8798f57555cb4152f9b2e1d0 where they "fixed" a leak that breaks our code. What happens is that at some point we invoke [this function](https://github.com/python/cpython/blob/97bfe8d3ebb0a54c8798f57555cb4152f9b2e1d0/Lib/multiprocessing/pool.py#L241-L258) with `ctx` being:
```
<multiprocessing.context.ForkContext object at 0x7f25d463fcf8>
```
then in the `__init__` method of `Process` we have [this assertion](https://github.com/python/cpython/blob/75d9d59ab3a372d3d78e6a1f5e9f256e29d0a9a6/Lib/multiprocessing/process.py#L74). Currently I am not sure if the bug was latent on our side, or was introduce by the commit above in `multiprocessing`.
Looking at the documentation of the `multiprocessing` module this seems to be a bug on our side, in the sense that the code in `web.py` depends on implementation details of `multiprocessing` that have been refactored extensively.
An [interesting thread](https://mail.python.org/pipermail/python-list/2011-March/600070.html) on the subject of `daemon` processes in `multiprocessing`.
Will submit a PR shortly.

Resolving patch:
diff --git a/lib/spack/spack/util/web.py b/lib/spack/spack/util/web.py
--- a/lib/spack/spack/util/web.py
+++ b/lib/spack/spack/util/web.py
@@ -60,18 +60,30 @@ def handle_starttag(self, tag, attrs):
 
 class NonDaemonProcess(multiprocessing.Process):
     """Process tha allows sub-processes, so pools can have sub-pools."""
-    def _get_daemon(self):
+    @property
+    def daemon(self):
         return False
 
-    def _set_daemon(self, value):
+    @daemon.setter
+    def daemon(self, value):
         pass
 
-    daemon = property(_get_daemon, _set_daemon)
 
+if sys.version_info[0] < 3:
+    class NonDaemonPool(multiprocessing.pool.Pool):
+        """Pool that uses non-daemon processes"""
+        Process = NonDaemonProcess
+else:
 
-class NonDaemonPool(multiprocessing.pool.Pool):
-    """Pool that uses non-daemon processes"""
-    Process = NonDaemonProcess
+    class NonDaemonContext(type(multiprocessing.get_context())):
+        Process = NonDaemonProcess
+
+    class NonDaemonPool(multiprocessing.pool.Pool):
+        """Pool that uses non-daemon processes"""
+
+        def __init__(self, *args, **kwargs):
+            kwargs['context'] = NonDaemonContext()
+            super(NonDaemonPool, self).__init__(*args, **kwargs)
 
 
 def _spider(url, visited, root, depth, max_depth, raise_on_error):
@@ -310,7 +322,7 @@ def find_versions_of_archive(archive_urls, list_url=None, list_depth=0):
         #   .sha256
         #   .sig
         # However, SourceForge downloads still need to end in '/download'.
-        url_regex += '(\/download)?$'
+        url_regex += r'(\/download)?$'
 
         regexes.append(url_regex)
 


Test patch: