Repository: spack/spack

Issue:
_urlopen purges 'context' from request, causing SSL no-verify to be ignored (spack -k)
### Steps to reproduce

urllib is the default url retriever for http and https protocol, however, it is not possible to retrieve any packages using `spack -k` (disabling SSL verification) with urllib (bug introduced by deleting `context` ). There is a bug in lib/spack/spack/util/web.py because it cannot behave properly with the `-k` flag for Spack.

Please examine lib/spack/spack/util/web.py:

- Note that `context` is removed from kwargs for all calls to `_urlopen` in order to be backwards compatible with urllib on line 514 (urrlib functionality varies from version of Python to version of Python, with a comment on line 511)
- `context` is needed in kwargs if VerifyMode.CERT_NONE is set (which will be the case if `spack -k` is used) otherwise `opener(req,...)` will have a failed to verify SSL error (reference line 108).

This combination of setting a context and then intentionally removing it means that `spack -k` will fail to work as intended for http and https protocol.

Reproduction:
- Be on a machine where SSL verification will fail (behind a proxy is a good place to start), use the `-k` flag with Spack to see that all packages using the `git` protocol fetch and succeed. Try to install `cmake` as an example with `spack -k install cmake` and the fetcher will fail. 

This relates to issue #25254 and is due to switching from curl as default to urllib


### Error message

```==> [2021-08-09-07:04:22.649134] FailedDownloadError: Failed to fetch file from URL: https://mirror.spack.io/_source-cache/archive/fa/fac3915171d4dff25913975d712f76e69aef44bf738ba7b976793a458b4cfed4.tar.gz
    Urllib fetch failed to verify url https://mirror.spack.io/_source-cache/archive/fa/fac3915171d4dff25913975d712f76e69aef44bf738ba7b976793a458b4cfed4.tar.gz
==> [2021-08-09-07:04:22.652664] Checking existence of https://mirror.spack.io/cmake/cmake-3.21.1.tar.gz
urllib
https://mirror.spack.io/cmake/cmake-3.21.1.tar.gz
here2
==> [2021-08-09-07:05:02.750286] FailedDownloadError: Failed to fetch file from URL: https://mirror.spack.io/cmake/cmake-3.21.1.tar.gz
    Urllib fetch failed to verify url https://mirror.spack.io/cmake/cmake-3.21.1.tar.gz
==> [2021-08-09-07:05:02.753044] Checking existence of https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1.tar.gz
urllib
https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1.tar.gz
here2
==> [2021-08-09-07:05:12.813018] FailedDownloadError: Failed to fetch file from URL: https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1.tar.gz
    Urllib fetch failed to verify url https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1.tar.gz
==> [2021-08-09-07:05:12.816336] Fetching from https://mirror.spack.io/_source-cache/archive/fa/fac3915171d4dff25913975d712f76e69aef44bf738ba7b976793a458b4cfed4.tar.gz failed.
==> [2021-08-09-07:05:12.817188] Fetching from https://mirror.spack.io/cmake/cmake-3.21.1.tar.gz failed.
==> [2021-08-09-07:05:12.817991] Fetching from https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1.tar.gz failed.
==> [2021-08-09-07:05:12.832977] Error: FetchError: All fetchers failed

/Users/pakuber/test_spack_xyce/spack/lib/spack/spack/package.py:1379, in do_fetch:
       1376
       1377        self.stage.create()
       1378        err_msg = None if not self.manual_download else self.download_instr
  >>   1379        start_time = time.time()
       1380        self.stage.fetch(mirror_only, err_msg=err_msg)
       1381        self._fetch_time = time.time() - start_time
       1382
```

### Information on your system

* **Spack:** 0.16.2-3864-a3f76740e7
* **Python:** 3.7.8
* **Platform:** darwin-catalina-skylake
* **Concretizer:** original

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:

Resolving patch:
diff --git a/lib/spack/spack/util/web.py b/lib/spack/spack/util/web.py
--- a/lib/spack/spack/util/web.py
+++ b/lib/spack/spack/util/web.py
@@ -508,9 +508,9 @@ def _urlopen(req, *args, **kwargs):
     except AttributeError:
         pass
 
-    # We don't pass 'context' parameter because it was only introduced starting
+    # Note: 'context' parameter was only introduced starting
     # with versions 2.7.9 and 3.4.3 of Python.
-    if 'context' in kwargs:
+    if __UNABLE_TO_VERIFY_SSL:
         del kwargs['context']
 
     opener = urlopen
@@ -518,7 +518,13 @@ def _urlopen(req, *args, **kwargs):
         import spack.s3_handler
         opener = spack.s3_handler.open
 
-    return opener(req, *args, **kwargs)
+    try:
+        return opener(req, *args, **kwargs)
+    except TypeError as err:
+        # If the above fails because of 'context', call without 'context'.
+        if 'context' in kwargs and 'context' in str(err):
+            del kwargs['context']
+        return opener(req, *args, **kwargs)
 
 
 def find_versions_of_archive(


Test patch: