Repository: spack/spack

Issue:
Python 3.9.6: Error calling urllib.parse
With Python 3.9.6, [this change](https://github.com/python/cpython/blob/db3ff76da19004f266b62e98a81bdfd322861436/Lib/urllib/parse.py#L462) in `urllib.parse` causes an error using `urllib.parse.urlparse(..., scheme=None)` like here: https://github.com/spack/spack/blob/59028aa0a5d4f106c33955a9779ea3ae19864dd9/lib/spack/spack/util/url.py#L167-L168

I don't think this is a Python bug since the [docs](https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urlparse) state:

> [scheme] should be the same type (text or bytes) as urlstring

However, [typeshed](https://github.com/python/typeshed/blob/c6b78354e6b9f979bb5b83044060b67e9c55a0ae/stdlib/urllib/parse.pyi#L125-L126) does have it typed as `Optional[str]` so I'm not sure.


### Steps to reproduce the issue

```console
$ python3 --version
Python 3.9.6
$ spack install zlib
==> Installing zlib-1.2.11-74mwnxgn6nujehpyyalhwizwojwn5zga
==> Error: Failed to install zlib due to AttributeError: 'NoneType' object has no attribute 'replace'
==> Error: 'NoneType' object has no attribute 'replace'
$ python3 -c "from urllib.parse import urlparse; urlparse('', scheme=None)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/parse.py", line 393, in urlparse
    splitresult = urlsplit(url, scheme, allow_fragments)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/parse.py", line 462, in urlsplit
    scheme = scheme.replace(b, "")
AttributeError: 'NoneType' object has no attribute 'replace'
```

### Error Message

<details>
  <summary>Stacktrace</summary>

```console
$ spack --debug --stacktrace install zlib
Cellar/spack/0.16.2/lib/spack/spack/cmd/__init__.py:122 ==> [2021-06-30-19:23:27.332263] Imported install from built-in commands
Cellar/spack/0.16.2/lib/spack/spack/config.py:905 ==> [2021-06-30-19:23:27.336600] Reading config file /usr/local/Cellar/spack/0.16.2/etc/spack/defaults/config.yaml
Cellar/spack/0.16.2/lib/spack/spack/cmd/__init__.py:122 ==> [2021-06-30-19:23:27.370003] Imported install from built-in commands
Cellar/spack/0.16.2/lib/spack/spack/config.py:905 ==> [2021-06-30-19:23:27.379310] Reading config file /usr/local/Cellar/spack/0.16.2/etc/spack/defaults/repos.yaml
Cellar/spack/0.16.2/lib/spack/spack/config.py:905 ==> [2021-06-30-19:23:27.431903] Reading config file /usr/local/Cellar/spack/0.16.2/etc/spack/defaults/packages.yaml
Cellar/spack/0.16.2/lib/spack/spack/config.py:905 ==> [2021-06-30-19:23:27.463903] Reading config file /usr/local/Cellar/spack/0.16.2/etc/spack/defaults/darwin/packages.yaml
Cellar/spack/0.16.2/lib/spack/spack/config.py:905 ==> [2021-06-30-19:23:27.473058] Reading config file /Users/Branch/.spack/darwin/compilers.yaml
Cellar/spack/0.16.2/lib/spack/spack/database.py:361 ==> [2021-06-30-19:23:27.525082] DATABASE LOCK TIMEOUT: 3s
Cellar/spack/0.16.2/lib/spack/spack/database.py:365 ==> [2021-06-30-19:23:27.525475] PACKAGE LOCK TIMEOUT: No timeout
Cellar/spack/0.16.2/lib/spack/spack/installer.py:1371 ==> [2021-06-30-19:23:27.532093] Initializing the build queue from the build requests
Cellar/spack/0.16.2/lib/spack/spack/installer.py:989 ==> [2021-06-30-19:23:27.532547] Initializing the build queue for zlib
Cellar/spack/0.16.2/lib/spack/spack/installer.py:2054 ==> [2021-06-30-19:23:27.533780] Processing dependencies for zlib-1.2.11-74mwnxgn6nujehpyyalhwizwojwn5zga: ('build', 'link', 'run')
Cellar/spack/0.16.2/lib/spack/spack/database.py:461 ==> [2021-06-30-19:23:27.553180] Removing failure marking for zlib
Cellar/spack/0.16.2/lib/spack/spack/installer.py:1793 ==> [2021-06-30-19:23:27.553894] Pkg id zlib-1.2.11-74mwnxgn6nujehpyyalhwizwojwn5zga has the following dependents:
Cellar/spack/0.16.2/lib/spack/spack/installer.py:1377 ==> [2021-06-30-19:23:27.555060] Ensure all dependencies know all dependents across specs
Cellar/spack/0.16.2/lib/spack/spack/installer.py:935 ==> [2021-06-30-19:23:27.556576] Acquiring a write lock on zlib-1.2.11-74mwnxgn6nujehpyyalhwizwojwn5zga with timeout 1e-09
Cellar/spack/0.16.2/lib/spack/spack/stage.py:312 ==> [2021-06-30-19:23:27.561416] Creating stage lock spack-stage-zlib-1.2.11-74mwnxgn6nujehpyyalhwizwojwn5zga
Cellar/spack/0.16.2/lib/spack/spack/installer.py:1095 ==> [2021-06-30-19:23:27.563466] Installing zlib-1.2.11-74mwnxgn6nujehpyyalhwizwojwn5zga
Cellar/spack/0.16.2/lib/spack/spack/installer.py:393 ==> [2021-06-30-19:23:27.563803] Searching for binary cache of zlib-1.2.11-74mwnxgn6nujehpyyalhwizwojwn5zga
Cellar/spack/0.16.2/lib/spack/spack/config.py:905 ==> [2021-06-30-19:23:27.565923] Reading config file /usr/local/Cellar/spack/0.16.2/etc/spack/defaults/mirrors.yaml
Cellar/spack/0.16.2/lib/spack/spack/installer.py:1301 ==> [2021-06-30-19:23:27.569783] Flagging zlib-1.2.11-74mwnxgn6nujehpyyalhwizwojwn5zga as failed: 'NoneType' object has no attribute 'replace'
Cellar/spack/0.16.2/lib/spack/spack/installer.py:1577 ==> [2021-06-30-19:23:27.571197] Error: Failed to install zlib due to AttributeError: 'NoneType' object has no attribute 'replace'
Traceback (most recent call last):
  File "/usr/local/bin/spack", line 66, in <module>
    sys.exit(spack.main.main())
  File "/usr/local/Cellar/spack/0.16.2/lib/spack/spack/main.py", line 762, in main
    return _invoke_command(command, parser, args, unknown)
  File "/usr/local/Cellar/spack/0.16.2/lib/spack/spack/main.py", line 490, in _invoke_command
    return_val = command(parser, args)
  File "/usr/local/Cellar/spack/0.16.2/lib/spack/spack/cmd/install.py", line 365, in install
    install_specs(args, kwargs, zip(abstract_specs, specs))
  File "/usr/local/Cellar/spack/0.16.2/lib/spack/spack/cmd/install.py", line 207, in install_specs
    builder.install()
  File "/usr/local/Cellar/spack/0.16.2/lib/spack/spack/installer.py", line 1535, in install
    self._install_task(task)
  File "/usr/local/Cellar/spack/0.16.2/lib/spack/spack/installer.py", line 1101, in _install_task
    _install_from_cache(pkg, cache_only, explicit, unsigned,
  File "/usr/local/Cellar/spack/0.16.2/lib/spack/spack/installer.py", line 273, in _install_from_cache
    installed_from_cache = _try_install_from_binary_cache(
  File "/usr/local/Cellar/spack/0.16.2/lib/spack/spack/installer.py", line 394, in _try_install_from_binary_cache
    matches = binary_distribution.get_mirrors_for_spec(
  File "/usr/local/Cellar/spack/0.16.2/lib/spack/spack/binary_distribution.py", line 1396, in get_mirrors_for_spec
    results = try_direct_fetch(spec,
  File "/usr/local/Cellar/spack/0.16.2/lib/spack/spack/binary_distribution.py", line 1336, in try_direct_fetch
    buildcache_fetch_url = url_util.join(
  File "/usr/local/Cellar/spack/0.16.2/lib/spack/spack/util/url.py", line 156, in join
    obj = urllib_parse.urlparse(
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/parse.py", line 393, in urlparse
    splitresult = urlsplit(url, scheme, allow_fragments)
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/parse.py", line 462, in urlsplit
    scheme = scheme.replace(b, "")
AttributeError: 'NoneType' object has no attribute 'replace'
```
</details>

### Information on your system

* **Spack:** 0.16.2
* **Python:** 3.9.6
* **Platform:** darwin-bigsur-skylake

### Additional information

- [x] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have searched the issues of this repo and believe this is not a duplicate
- [x] I have run the failing commands in debug mode and reported the output




Discussion:
Thanks for reporting this @branchvincent, didn't realize Homebrew even had a package for Spack!

The use of `urllib` was recently added in #21398, maybe @loulawrence can take a look at this failure. Not sure if this is a bug in Python or not, but we can definitely figure out some kind of workaround. Also, it's sad that Homebrew's CI caught this bug but our CI didn't. We should investigate that as well, since we also test on macOS with Python 3.9.

Resolving patch:
diff --git a/lib/spack/spack/util/url.py b/lib/spack/spack/util/url.py
--- a/lib/spack/spack/util/url.py
+++ b/lib/spack/spack/util/url.py
@@ -11,8 +11,8 @@
 import os.path
 import re
 
-from six import string_types
 import six.moves.urllib.parse as urllib_parse
+from six import string_types
 
 import spack.util.path
 
@@ -151,21 +151,21 @@ def join(base_url, path, *extra, **kwargs):
         for x in itertools.chain((base_url, path), extra)]
     n = len(paths)
     last_abs_component = None
-    scheme = None
+    scheme = ''
     for i in range(n - 1, -1, -1):
         obj = urllib_parse.urlparse(
-            paths[i], scheme=None, allow_fragments=False)
+            paths[i], scheme='', allow_fragments=False)
 
         scheme = obj.scheme
 
         # in either case the component is absolute
-        if scheme is not None or obj.path.startswith('/'):
-            if scheme is None:
+        if scheme or obj.path.startswith('/'):
+            if not scheme:
                 # Without a scheme, we have to go back looking for the
                 # next-last component that specifies a scheme.
                 for j in range(i - 1, -1, -1):
                     obj = urllib_parse.urlparse(
-                        paths[j], scheme=None, allow_fragments=False)
+                        paths[j], scheme='', allow_fragments=False)
 
                     if obj.scheme:
                         paths[i] = '{SM}://{NL}{PATH}'.format(


Test patch: