Repository: spack/spack

Issue:
`buildcache push`: cryptic error when required OCI credentials are missing
### Steps to reproduce

When trying to push to a (OCI) build cache that there are no credentials for but are required, the last line of an unhelpful traceback pops up, seemingly because of trying to read from an unpopulated string buffer.

`spack.yaml`:
```yaml
spack:
  view: my_view
  specs:
  - git
  config:
    install_tree:
      # root: /opt/spack
      padded_length: 128

  mirrors:
    local-buildcache:
      url: oci://ghcr.io/berquist/spack-gha-buildcache-example
      # push:
      #   access_pair:
      #   - berquist
      #   - $GITHUB_TOKEN
```
```bash
cd /dir/with/spack.yaml
spack env activate .
spack concretize --fresh --force
spack install --fail-fast
spack --debug buildcache push --unsigned local-buildcache >& spack.log
```

### Error message

```
==> [2023-12-22-11:02:15.955861] Reading config from file /home/eric/data/spack/etc/spack/defaults/config.yaml
==> [2023-12-22-11:02:15.970909] Reading config from file /home/eric/.spack/config.yaml
==> [2023-12-22-11:02:15.972265] Reading config from file /home/eric/development/spack-gha-buildcache-example/spack.yaml
==> [2023-12-22-11:02:15.975399] Reading config from file /home/eric/data/spack/etc/spack/defaults/config.yaml
==> [2023-12-22-11:02:15.989858] Reading config from file /home/eric/.spack/config.yaml
==> [2023-12-22-11:02:15.991115] Using environment '/home/eric/development/spack-gha-buildcache-example'
==> [2023-12-22-11:02:15.995112] Imported buildcache from built-in commands
==> [2023-12-22-11:02:15.998133] Imported buildcache from built-in commands
==> [2023-12-22-11:02:15.998554] Reading config from file /home/eric/data/spack/etc/spack/defaults/mirrors.yaml
==> [2023-12-22-11:02:16.000147] Reading config from file /home/eric/.spack/mirrors.yaml
==> [2023-12-22-11:02:16.001925] DATABASE LOCK TIMEOUT: 60s
==> [2023-12-22-11:02:16.001981] PACKAGE LOCK TIMEOUT: No timeout
==> [2023-12-22-11:02:16.012151] Selected 40 specs to push to oci://ghcr.io/berquist/spack-gha-buildcache-example
==> [2023-12-22-11:02:16.039281] Checking for existing specs in the buildcache
<note: not 40 because of previously pushing zstd + dependencies>
==> [2023-12-22-11:02:16.728342] 37 specs need to be pushed to ghcr.io/berquist/spack-gha-buildcache-example
==> [2023-12-22-11:02:16.742231] '/bin/file' '-b' '-h' '--mime-type' '/home/eric/data/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_pl/linux-archrolling-zen/gcc-13.2.1/readline-8.2-36iv5uy6bafh3wjxtvmgownsmqnkh6ga/include/readline/chardefs.h'
<snip>
==> [2023-12-22-11:02:16.451539] Using cached archive: /home/eric/data/spack/var/spack/cache/blobs/sha256/64aaff9941a5aa35242f52376ecf7e2c786a99aca42f1b747be3146d6d193bf9
Traceback (most recent call last):
  File "/home/eric/data/spack/bin/spack", line 52, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/eric/data/spack/lib/spack/spack_installable/main.py", line 42, in main
    sys.exit(spack.main.main(argv))
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/eric/data/spack/lib/spack/spack/main.py", line 1066, in main
    return _main(argv)
           ^^^^^^^^^^^
  File "/home/eric/data/spack/lib/spack/spack/main.py", line 1019, in _main
    return finish_parse_and_run(parser, cmd_name, args, env_format_error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eric/data/spack/lib/spack/spack/main.py", line 1049, in finish_parse_and_run
    return _invoke_command(command, parser, args, unknown)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eric/data/spack/lib/spack/spack/main.py", line 648, in _invoke_command
    return_val = command(parser, args)
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/home/eric/data/spack/lib/spack/spack/cmd/buildcache.py", line 1143, in buildcache
    args.func(args)
  File "/home/eric/data/spack/lib/spack/spack/cmd/buildcache.py", line 397, in push_fn
    skipped, base_images, checksums = _push_oci(
                                      ^^^^^^^^^^
  File "/home/eric/data/spack/lib/spack/spack/cmd/buildcache.py", line 721, in _push_oci
    new_blobs = pool.starmap(
                ^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 375, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 774, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f061e57bf90>'. Reason: 'TypeError("cannot pickle '_io.BufferedReader' object")'

```

### Information on your system

* **Spack:** 0.22.0.dev0 (4540980337f6b516b634f075a82e515b3464cf6b)
* **Python:** 3.11.6
* **Platform:** linux-archrolling-zen
* **Concretizer:** clingo

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
I get a much better error
```
urllib.error.HTTPError: HTTP Error 401: Failed to login to https://ghcr.io/v2/berquist/spack-gha-buildcache-example/blobs/uploads/: Unauthorized
```
after
```diff
diff --git lib/spack/spack/cmd/buildcache.py lib/spack/spack/cmd/buildcache.py
index 75011a3e9b..9a13b2cb65 100644
--- lib/spack/spack/cmd/buildcache.py
+++ lib/spack/spack/cmd/buildcache.py
@@ -718,9 +718,7 @@ def _push_oci(
     )

     # Upload blobs
-    new_blobs = pool.starmap(
-        _push_single_spack_binary_blob, ((target_image, spec, tmpdir) for spec in to_be_uploaded)
-    )
+    new_blobs = [_push_single_spack_binary_blob(target_image, spec, tmpdir) for spec in to_be_uploaded]

     # And update the spec to blob mapping
     for spec, blob in zip(to_be_uploaded, new_blobs):
```
so the multiprocessing code is obscuring it.
@haampie ^

Resolving patch:
diff --git a/lib/spack/spack/cmd/buildcache.py b/lib/spack/spack/cmd/buildcache.py
--- a/lib/spack/spack/cmd/buildcache.py
+++ b/lib/spack/spack/cmd/buildcache.py
@@ -803,8 +803,9 @@ def _config_from_tag(image_ref: ImageReference, tag: str) -> Optional[dict]:
 
 
 def _update_index_oci(image_ref: ImageReference, tmpdir: str, pool: MaybePool) -> None:
-    response = spack.oci.opener.urlopen(urllib.request.Request(url=image_ref.tags_url()))
-    spack.oci.opener.ensure_status(response, 200)
+    request = urllib.request.Request(url=image_ref.tags_url())
+    response = spack.oci.opener.urlopen(request)
+    spack.oci.opener.ensure_status(request, response, 200)
     tags = json.load(response)["tags"]
 
     # Fetch all image config files in parallel
diff --git a/lib/spack/spack/oci/oci.py b/lib/spack/spack/oci/oci.py
--- a/lib/spack/spack/oci/oci.py
+++ b/lib/spack/spack/oci/oci.py
@@ -134,7 +134,7 @@ def upload_blob(
             return True
 
         # Otherwise, do another PUT request.
-        spack.oci.opener.ensure_status(response, 202)
+        spack.oci.opener.ensure_status(request, response, 202)
         assert "Location" in response.headers
 
         # Can be absolute or relative, joining handles both
@@ -143,19 +143,16 @@ def upload_blob(
         )
         f.seek(0)
 
-        response = _urlopen(
-            Request(
-                url=upload_url,
-                method="PUT",
-                data=f,
-                headers={
-                    "Content-Type": "application/octet-stream",
-                    "Content-Length": str(file_size),
-                },
-            )
+        request = Request(
+            url=upload_url,
+            method="PUT",
+            data=f,
+            headers={"Content-Type": "application/octet-stream", "Content-Length": str(file_size)},
         )
 
-        spack.oci.opener.ensure_status(response, 201)
+        response = _urlopen(request)
+
+        spack.oci.opener.ensure_status(request, response, 201)
 
     # print elapsed time and # MB/s
     _log_upload_progress(digest, file_size, time.time() - start)
@@ -189,16 +186,16 @@ def upload_manifest(
     if not tag:
         ref = ref.with_digest(digest)
 
-    response = _urlopen(
-        Request(
-            url=ref.manifest_url(),
-            method="PUT",
-            data=data,
-            headers={"Content-Type": oci_manifest["mediaType"]},
-        )
+    request = Request(
+        url=ref.manifest_url(),
+        method="PUT",
+        data=data,
+        headers={"Content-Type": oci_manifest["mediaType"]},
     )
 
-    spack.oci.opener.ensure_status(response, 201)
+    response = _urlopen(request)
+
+    spack.oci.opener.ensure_status(request, response, 201)
     return digest, size
 
 
diff --git a/lib/spack/spack/oci/opener.py b/lib/spack/spack/oci/opener.py
--- a/lib/spack/spack/oci/opener.py
+++ b/lib/spack/spack/oci/opener.py
@@ -310,19 +310,15 @@ def http_error_401(self, req: Request, fp, code, msg, headers):
         # Login failed, avoid infinite recursion where we go back and
         # forth between auth server and registry
         if hasattr(req, "login_attempted"):
-            raise urllib.error.HTTPError(
-                req.full_url, code, f"Failed to login to {req.full_url}: {msg}", headers, fp
+            raise spack.util.web.DetailedHTTPError(
+                req, code, f"Failed to login: {msg}", headers, fp
             )
 
         # On 401 Unauthorized, parse the WWW-Authenticate header
         # to determine what authentication is required
         if "WWW-Authenticate" not in headers:
-            raise urllib.error.HTTPError(
-                req.full_url,
-                code,
-                "Cannot login to registry, missing WWW-Authenticate header",
-                headers,
-                fp,
+            raise spack.util.web.DetailedHTTPError(
+                req, code, "Cannot login to registry, missing WWW-Authenticate header", headers, fp
             )
 
         header_value = headers["WWW-Authenticate"]
@@ -330,8 +326,8 @@ def http_error_401(self, req: Request, fp, code, msg, headers):
         try:
             challenge = get_bearer_challenge(parse_www_authenticate(header_value))
         except ValueError as e:
-            raise urllib.error.HTTPError(
-                req.full_url,
+            raise spack.util.web.DetailedHTTPError(
+                req,
                 code,
                 f"Cannot login to registry, malformed WWW-Authenticate header: {header_value}",
                 headers,
@@ -340,8 +336,8 @@ def http_error_401(self, req: Request, fp, code, msg, headers):
 
         # If there is no bearer challenge, we can't handle it
         if not challenge:
-            raise urllib.error.HTTPError(
-                req.full_url,
+            raise spack.util.web.DetailedHTTPError(
+                req,
                 code,
                 f"Cannot login to registry, unsupported authentication scheme: {header_value}",
                 headers,
@@ -356,8 +352,8 @@ def http_error_401(self, req: Request, fp, code, msg, headers):
                 timeout=req.timeout,
             )
         except ValueError as e:
-            raise urllib.error.HTTPError(
-                req.full_url,
+            raise spack.util.web.DetailedHTTPError(
+                req,
                 code,
                 f"Cannot login to registry, failed to obtain bearer token: {e}",
                 headers,
@@ -412,13 +408,13 @@ def create_opener():
     return opener
 
 
-def ensure_status(response: HTTPResponse, status: int):
+def ensure_status(request: urllib.request.Request, response: HTTPResponse, status: int):
     """Raise an error if the response status is not the expected one."""
     if response.status == status:
         return
 
-    raise urllib.error.HTTPError(
-        response.geturl(), response.status, response.reason, response.info(), None
+    raise spack.util.web.DetailedHTTPError(
+        request, response.status, response.reason, response.info(), None
     )
 
 


Test patch: