Repository: spack/spack

Issue:
Out of date external six breaks builtin urllib3
### Steps to reproduce

```console
$ spack mirror add my-spackages s3://my-spackages
$ spack -d buildcache create -m my-spackages pcl
```

After some debugging of urllib3 reporting a parse error in what appeared to be a well-formed URL, I modified the system `urllib3.util.url.py` as follows to obtain a full backtrace. In `parse_url`:
```python3
-    except (ValueError, AttributeError):
-        return six.raise_from(LocationParseError(source_url), None)
+    except (ValueError, AttributeError) as parse_exc:
+        return six.raise_from(LocationParseError(source_url), parse_exc)
```

This revealed the root cause as the `AttributeError: module 'six' has no attribute 'ensure_str'` in the backtrace below. 
I also found that invoking `parse_url("https://my-spackages.s3.amazonaws.com/build_cache/linux-ubuntu20.04-skylake_avx512/gcc-9.3.0/pcl-develop/linux-ubuntu20.04-skylake_avx512-gcc-9.3.0-pcl-develop-xquo4jquuxeqqfocqih7uknmavdctzgn.spack")` from the system's Python 3 repl parsed the same URL, copied and pasted from the log, as expected.

Googling turned up [this Debian issue](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=952591), which lead me to compare `six.__file__` between the system REPL and dropping into a console from the debugger while running spack.

### Error message

```console
$ spack -d buildcache create -m my-spackages pcl
==> [2021-11-15-20:35:41.587552] Reading config file /home/ubuntu/spack/etc/spack/defaults/config.yaml
==> [2021-11-15-20:35:41.803853] Reading config file /home/ubuntu/spack/var/spack/environments/my-env/spack.yaml
==> [2021-11-15-20:35:41.948654] Using environment 'my-env'
==> [2021-11-15-20:35:41.951370] Imported buildcache from built-in commands
==> [2021-11-15-20:35:41.960879] Imported buildcache from built-in commands
==> [2021-11-15-20:35:41.962268] Reading config file /home/ubuntu/spack/etc/spack/defaults/mirrors.yaml
==> [2021-11-15-20:35:41.969796] find_matching_specs: about to parse specs for ['pcl']
==> [2021-11-15-20:35:41.970977] Reading config file /home/ubuntu/spack/etc/spack/defaults/bootstrap.yaml
==> [2021-11-15-20:35:41.994821] DATABASE LOCK TIMEOUT: 3s
==> [2021-11-15-20:35:41.995189] PACKAGE LOCK TIMEOUT: No timeout
==> [2021-11-15-20:35:42.011842] Reading config file /home/ubuntu/spack/etc/spack/defaults/repos.yaml
==> [2021-11-15-20:35:42.016536] Reading config file /home/ubuntu/.spack/repos.yaml
==> [2021-11-15-20:35:42.132395] Buildcache files will be output to s3://my-spackages/build_cache
==> [2021-11-15-20:35:42.132640] Found at least one matching spec
==> [2021-11-15-20:35:42.134275] examining match pcl@develop%gcc@9.3.0~ipo build_type=RelWithDebInfo arch=linux-ubuntu20.04-skylake_avx512
==> [2021-11-15-20:35:42.136222] adding matching spec pcl@develop%gcc@9.3.0~ipo build_type=RelWithDebInfo arch=linux-ubuntu20.04-skylake_avx512
==> [2021-11-15-20:35:42.136529] recursing dependencies
==> [2021-11-15-20:35:42.137903] writing tarballs to s3://my-spackages/build_cache
==> [2021-11-15-20:35:42.138837] creating binary cache file for package pcl@develop%gcc@9.3.0~ipo build_type=RelWithDebInfo arch=linux-ubuntu20.04-skylake_avx512 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/util/url.py", line 381, in parse_url
    host = _normalize_host(host, scheme)
  File "/usr/lib/python3/dist-packages/urllib3/util/url.py", line 296, in _normalize_host
    return six.ensure_str(
AttributeError: module 'six' has no attribute 'ensure_str'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/httpsession.py", line 373, in send
    conn = manager.connection_from_url(request.url)
  File "/usr/lib/python3/dist-packages/urllib3/poolmanager.py", line 284, in connection_from_url
    u = parse_url(url)
  File "/usr/lib/python3/dist-packages/urllib3/util/url.py", line 392, in parse_url
    return six.raise_from(LocationParseError(source_url), parse_exc)
  File "<string>", line 3, in raise_from
urllib3.exceptions.LocationParseError: Failed to parse: https://my-spackages.s3.amazonaws.com/build_cache/linux-ubuntu20.04-skylake_avx512/gcc-9.3.0/pcl-develop/linux-ubuntu20.04-skylake_avx512-gcc-9.3.0-pcl-develop-xquo4jquuxeqqfocqih7uknmavdctzgn.spack

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/spack/bin/spack", line 100, in <module>
    sys.exit(spack.main.main())
  File "/home/ubuntu/spack/lib/spack/spack/main.py", line 882, in main
    return _main(argv)
  File "/home/ubuntu/spack/lib/spack/spack/main.py", line 865, in _main
    return _invoke_command(command, parser, args, unknown)
  File "/home/ubuntu/spack/lib/spack/spack/main.py", line 535, in _invoke_command
    return_val = command(parser, args)
  File "/home/ubuntu/spack/lib/spack/spack/cmd/buildcache.py", line 993, in buildcache
    args.func(args)
  File "/home/ubuntu/spack/lib/spack/spack/cmd/buildcache.py", line 514, in createtarball
    _createtarball(env, spec_file=args.spec_file, packages=args.specs,
  File "/home/ubuntu/spack/lib/spack/spack/cmd/buildcache.py", line 463, in _createtarball
    bindist.build_tarball(spec, outdir, force, make_relative,
  File "/home/ubuntu/spack/lib/spack/spack/binary_distribution.py", line 1001, in build_tarball
    if web_util.url_exists(remote_spackfile_path):
  File "/home/ubuntu/spack/lib/spack/spack/util/web.py", line 229, in url_exists
    s3.get_object(Bucket=url.netloc, Key=url.path.lstrip('/'))
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/client.py", line 705, in _make_api_call
    http, parsed_response = self._make_request(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/client.py", line 725, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/endpoint.py", line 136, in _send_request
    while self._needs_retry(attempts, operation_model, request_dict,
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/endpoint.py", line 252, in _needs_retry
    responses = self._event_emitter.emit(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/hooks.py", line 357, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/retryhandler.py", line 250, in __call__
    should_retry = self._should_retry(attempt_number, response,
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/retryhandler.py", line 269, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/retryhandler.py", line 316, in __call__
    checker_response = checker(attempt_number, response,
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/retryhandler.py", line 222, in __call__
    return self._check_caught_exception(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/endpoint.py", line 199, in _do_get_response
    http_response = self._send(request)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/endpoint.py", line 268, in _send
    return self.http_session.send(request)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/botocore/httpsession.py", line 431, in send
    raise HTTPClientError(error=e)
botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: Failed to parse: https://my-spackages.s3.amazonaws.com/build_cache/linux-ubuntu20.04-skylake_avx512/gcc-9.3.0/pcl-develop/linux-ubuntu20.04-skylake_avx512-gcc-9.3.0-pcl-develop-xquo4jquuxeqqfocqih7uknmavdctzgn.spack
```

### Information on your system

```console
* **Spack:** 0.17.0-76-0d06bf6027
* **Python:** 3.8.10
* **Platform:** linux-ubuntu20.04-skylake_avx512
* **Concretizer:** clingo
```

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:

Resolving patch:
diff --git a/lib/spack/external/__init__.py b/lib/spack/external/__init__.py
--- a/lib/spack/external/__init__.py
+++ b/lib/spack/external/__init__.py
@@ -103,7 +103,7 @@
 
 * Homepage: https://pypi.python.org/pypi/six
 * Usage: Python 2 and 3 compatibility utilities.
-* Version: 1.11.0
+* Version: 1.16.0
 
 macholib
 --------
diff --git a/lib/spack/external/six.py b/lib/spack/external/six.py
--- a/lib/spack/external/six.py
+++ b/lib/spack/external/six.py
@@ -1,4 +1,4 @@
-# Copyright (c) 2010-2017 Benjamin Peterson
+# Copyright (c) 2010-2020 Benjamin Peterson
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -29,7 +29,7 @@
 import types
 
 __author__ = "Benjamin Peterson <benjamin@python.org>"
-__version__ = "1.11.0"
+__version__ = "1.16.0"
 
 
 # Useful for very coarse version differentiation.
@@ -71,6 +71,11 @@ def __len__(self):
             MAXSIZE = int((1 << 63) - 1)
         del X
 
+if PY34:
+    from importlib.util import spec_from_loader
+else:
+    spec_from_loader = None
+
 
 def _add_doc(func, doc):
     """Add documentation to a function."""
@@ -186,6 +191,11 @@ def find_module(self, fullname, path=None):
             return self
         return None
 
+    def find_spec(self, fullname, path, target=None):
+        if fullname in self.known_modules:
+            return spec_from_loader(fullname, self)
+        return None
+
     def __get_module(self, fullname):
         try:
             return self.known_modules[fullname]
@@ -223,6 +233,12 @@ def get_code(self, fullname):
         return None
     get_source = get_code  # same as get_code
 
+    def create_module(self, spec):
+        return self.load_module(spec.name)
+
+    def exec_module(self, module):
+        pass
+
 _importer = _SixMetaPathImporter(__name__)
 
 
@@ -255,9 +271,11 @@ class _MovedItems(_LazyModule):
     MovedAttribute("zip_longest", "itertools", "itertools", "izip_longest", "zip_longest"),
     MovedModule("builtins", "__builtin__"),
     MovedModule("configparser", "ConfigParser"),
+    MovedModule("collections_abc", "collections", "collections.abc" if sys.version_info >= (3, 3) else "collections"),
     MovedModule("copyreg", "copy_reg"),
     MovedModule("dbm_gnu", "gdbm", "dbm.gnu"),
-    MovedModule("_dummy_thread", "dummy_thread", "_dummy_thread"),
+    MovedModule("dbm_ndbm", "dbm", "dbm.ndbm"),
+    MovedModule("_dummy_thread", "dummy_thread", "_dummy_thread" if sys.version_info < (3, 9) else "_thread"),
     MovedModule("http_cookiejar", "cookielib", "http.cookiejar"),
     MovedModule("http_cookies", "Cookie", "http.cookies"),
     MovedModule("html_entities", "htmlentitydefs", "html.entities"),
@@ -637,13 +655,16 @@ def u(s):
     import io
     StringIO = io.StringIO
     BytesIO = io.BytesIO
+    del io
     _assertCountEqual = "assertCountEqual"
     if sys.version_info[1] <= 1:
         _assertRaisesRegex = "assertRaisesRegexp"
         _assertRegex = "assertRegexpMatches"
+        _assertNotRegex = "assertNotRegexpMatches"
     else:
         _assertRaisesRegex = "assertRaisesRegex"
         _assertRegex = "assertRegex"
+        _assertNotRegex = "assertNotRegex"
 else:
     def b(s):
         return s
@@ -665,6 +686,7 @@ def indexbytes(buf, i):
     _assertCountEqual = "assertItemsEqual"
     _assertRaisesRegex = "assertRaisesRegexp"
     _assertRegex = "assertRegexpMatches"
+    _assertNotRegex = "assertNotRegexpMatches"
 _add_doc(b, """Byte literal""")
 _add_doc(u, """Text literal""")
 
@@ -681,6 +703,10 @@ def assertRegex(self, *args, **kwargs):
     return getattr(self, _assertRegex)(*args, **kwargs)
 
 
+def assertNotRegex(self, *args, **kwargs):
+    return getattr(self, _assertNotRegex)(*args, **kwargs)
+
+
 if PY3:
     exec_ = getattr(moves.builtins, "exec")
 
@@ -716,16 +742,7 @@ def exec_(_code_, _globs_=None, _locs_=None):
 """)
 
 
-if sys.version_info[:2] == (3, 2):
-    exec_("""def raise_from(value, from_value):
-    try:
-        if from_value is None:
-            raise value
-        raise value from from_value
-    finally:
-        value = None
-""")
-elif sys.version_info[:2] > (3, 2):
+if sys.version_info[:2] > (3,):
     exec_("""def raise_from(value, from_value):
     try:
         raise value from from_value
@@ -805,13 +822,33 @@ def print_(*args, **kwargs):
 _add_doc(reraise, """Reraise an exception.""")
 
 if sys.version_info[0:2] < (3, 4):
+    # This does exactly the same what the :func:`py3:functools.update_wrapper`
+    # function does on Python versions after 3.2. It sets the ``__wrapped__``
+    # attribute on ``wrapper`` object and it doesn't raise an error if any of
+    # the attributes mentioned in ``assigned`` and ``updated`` are missing on
+    # ``wrapped`` object.
+    def _update_wrapper(wrapper, wrapped,
+                        assigned=functools.WRAPPER_ASSIGNMENTS,
+                        updated=functools.WRAPPER_UPDATES):
+        for attr in assigned:
+            try:
+                value = getattr(wrapped, attr)
+            except AttributeError:
+                continue
+            else:
+                setattr(wrapper, attr, value)
+        for attr in updated:
+            getattr(wrapper, attr).update(getattr(wrapped, attr, {}))
+        wrapper.__wrapped__ = wrapped
+        return wrapper
+    _update_wrapper.__doc__ = functools.update_wrapper.__doc__
+
     def wraps(wrapped, assigned=functools.WRAPPER_ASSIGNMENTS,
               updated=functools.WRAPPER_UPDATES):
-        def wrapper(f):
-            f = functools.wraps(wrapped, assigned, updated)(f)
-            f.__wrapped__ = wrapped
-            return f
-        return wrapper
+        return functools.partial(_update_wrapper, wrapped=wrapped,
+                                 assigned=assigned, updated=updated)
+    wraps.__doc__ = functools.wraps.__doc__
+
 else:
     wraps = functools.wraps
 
@@ -824,7 +861,15 @@ def with_metaclass(meta, *bases):
     class metaclass(type):
 
         def __new__(cls, name, this_bases, d):
-            return meta(name, bases, d)
+            if sys.version_info[:2] >= (3, 7):
+                # This version introduced PEP 560 that requires a bit
+                # of extra care (we mimic what is done by __build_class__).
+                resolved_bases = types.resolve_bases(bases)
+                if resolved_bases is not bases:
+                    d['__orig_bases__'] = bases
+            else:
+                resolved_bases = bases
+            return meta(name, resolved_bases, d)
 
         @classmethod
         def __prepare__(cls, name, this_bases):
@@ -844,13 +889,75 @@ def wrapper(cls):
                 orig_vars.pop(slots_var)
         orig_vars.pop('__dict__', None)
         orig_vars.pop('__weakref__', None)
+        if hasattr(cls, '__qualname__'):
+            orig_vars['__qualname__'] = cls.__qualname__
         return metaclass(cls.__name__, cls.__bases__, orig_vars)
     return wrapper
 
 
+def ensure_binary(s, encoding='utf-8', errors='strict'):
+    """Coerce **s** to six.binary_type.
+
+    For Python 2:
+      - `unicode` -> encoded to `str`
+      - `str` -> `str`
+
+    For Python 3:
+      - `str` -> encoded to `bytes`
+      - `bytes` -> `bytes`
+    """
+    if isinstance(s, binary_type):
+        return s
+    if isinstance(s, text_type):
+        return s.encode(encoding, errors)
+    raise TypeError("not expecting type '%s'" % type(s))
+
+
+def ensure_str(s, encoding='utf-8', errors='strict'):
+    """Coerce *s* to `str`.
+
+    For Python 2:
+      - `unicode` -> encoded to `str`
+      - `str` -> `str`
+
+    For Python 3:
+      - `str` -> `str`
+      - `bytes` -> decoded to `str`
+    """
+    # Optimization: Fast return for the common case.
+    if type(s) is str:
+        return s
+    if PY2 and isinstance(s, text_type):
+        return s.encode(encoding, errors)
+    elif PY3 and isinstance(s, binary_type):
+        return s.decode(encoding, errors)
+    elif not isinstance(s, (text_type, binary_type)):
+        raise TypeError("not expecting type '%s'" % type(s))
+    return s
+
+
+def ensure_text(s, encoding='utf-8', errors='strict'):
+    """Coerce *s* to six.text_type.
+
+    For Python 2:
+      - `unicode` -> `unicode`
+      - `str` -> `unicode`
+
+    For Python 3:
+      - `str` -> `str`
+      - `bytes` -> decoded to `str`
+    """
+    if isinstance(s, binary_type):
+        return s.decode(encoding, errors)
+    elif isinstance(s, text_type):
+        return s
+    else:
+        raise TypeError("not expecting type '%s'" % type(s))
+
+
 def python_2_unicode_compatible(klass):
     """
-    A decorator that defines __unicode__ and __str__ methods under Python 2.
+    A class decorator that defines __unicode__ and __str__ methods under Python 2.
     Under Python 3 it does nothing.
 
     To support Python 2 and 3 with a single code base, define a __str__ method


Test patch: