Repository: canonical/charmcraft

Issue:
`fetch-libs` has weird error messages when libs aren't available
### Bug Description

Example: https://github.com/canonical/charmcraft/actions/runs/10049710901/job/27776909923#step:5:1228

When libraries aren't available from charmhub, `fetch-libs` doesn't always respond with a valid error message.

### To Reproduce

Use the `resources` spread test, but pointed at charmhub staging

### Environment

CI

### charmcraft.yaml

```shell
The output of `charmcraft init --profile flask-framework`
```


### Relevant log output

```shell
+ charmcraft fetch-libs
2024-07-22 23:34:31.157 Starting charmcraft, version 3.0.0.post96+g3b05ba5
2024-07-22 23:34:31.157 Logging execution to '/root/.local/state/charmcraft/log/charmcraft-20240722-233431.156756.log'
2024-07-22 23:34:31.157 Log verbosity level set to DEBUG
2024-07-22 23:34:31.157 Configuring application...
2024-07-22 23:34:31.157 Preparing application...
2024-07-22 23:34:31.158 Build plan: platform=None, build_for=None
2024-07-22 23:34:31.158 Loading project file '/charmcraft/tests/spread/commands/init-flask-framework/test-init/charmcraft.yaml'
2024-07-22 23:34:31.163 Processing grammar (on amd64 for amd64)
2024-07-22 23:34:31.163 Processing grammar for plugin: charm
2024-07-22 23:34:31.163 Processing grammar for source: .
2024-07-22 23:34:31.163 Processing grammar for build-snaps: ['rustup']
2024-07-22 23:34:31.165 Running charmcraft fetch-libs on host
2024-07-22 23:34:31.166 Setting up StoreService
2024-07-22 23:34:33.211 Falling back to file based storage
2024-07-22 23:34:33.212 Getting library metadata from charmhub
2024-07-22 23:34:33.213 HTTP 'POST' for 'https://api.staging.charmhub.io/v1/charm/libraries/bulk' with params None and headers {'User-Agent': 'charmcraft/3.0.0.post96+g3b05ba5 ubuntu/22.04 (x86_64) python/3.10.12'}
2024-07-22 23:34:34.588 charmcraft internal error: StoreServerError("Store operation failed:\n- None: Items need to include 'library_id' or 'package_id'")
2024-07-22 23:34:34.595 Traceback (most recent call last):
2024-07-22 23:34:34.596   File "/snap/charmcraft/x1/lib/python3.10/site-packages/craft_application/application.py", line 538, in run
2024-07-22 23:34:34.596     return_code = dispatcher.run() or 0
2024-07-22 23:34:34.596   File "/snap/charmcraft/x1/lib/python3.10/site-packages/craft_cli/dispatcher.py", line 487, in run
2024-07-22 23:34:34.596     return self._loaded_command.run(self._parsed_command_args)
2024-07-22 23:34:34.596   File "/snap/charmcraft/x1/lib/python3.10/site-packages/charmcraft/application/commands/store.py", line 1681, in run
2024-07-22 23:34:34.596     libs_metadata = store.get_libraries_metadata_by_name(charm_libs)
2024-07-22 23:34:34.596   File "/snap/charmcraft/x1/lib/python3.10/site-packages/charmcraft/services/store.py", line 248, in get_libraries_metadata_by_name
2024-07-22 23:34:34.596     for lib in self.get_libraries_metadata(libraries)
2024-07-22 23:34:34.596   File "/snap/charmcraft/x1/lib/python3.10/site-packages/charmcraft/services/store.py", line 240, in get_libraries_metadata
2024-07-22 23:34:34.597     return self.anonymous_client.fetch_libraries_metadata(store_libs)
2024-07-22 23:34:34.597   File "/snap/charmcraft/x1/lib/python3.10/site-packages/charmcraft/store/client.py", line 103, in fetch_libraries_metadata
2024-07-22 23:34:34.597     response = self.request_urlpath_json("POST", "/v1/charm/libraries/bulk", json=libs)
2024-07-22 23:34:34.597   File "/snap/charmcraft/x1/lib/python3.10/site-packages/charmcraft/store/client.py", line 64, in request_urlpath_json
2024-07-22 23:34:34.597     response = self._http_client.request(method, self.api_base_url + urlpath, *args, **kwargs)
2024-07-22 23:34:34.597   File "/snap/charmcraft/x1/lib/python3.10/site-packages/craft_store/http_client.py", line 166, in request
2024-07-22 23:34:34.597     raise errors.StoreServerError(response)
2024-07-22 23:34:34.597 craft_store.errors.StoreServerError: Store operation failed:
2024-07-22 23:34:34.597 - None: Items need to include 'library_id' or 'package_id'
2024-07-22 23:34:34.598 Full execution log: '/root/.local/state/charmcraft/log/charmcraft-20240722-233431.156756.log'
Traceback (most recent call last):
  File "/snap/charmcraft/x1/bin/charmcraft", line 8, in <module>
    sys.exit(main())
  File "/snap/charmcraft/x1/lib/python3.10/site-packages/charmcraft/application/main.py", line 160, in main
    return app.run()
  File "/snap/charmcraft/x1/lib/python3.10/site-packages/craft_application/application.py", line 538, in run
    return_code = dispatcher.run() or 0
  File "/snap/charmcraft/x1/lib/python3.10/site-packages/craft_cli/dispatcher.py", line 487, in run
    return self._loaded_command.run(self._parsed_command_args)
  File "/snap/charmcraft/x1/lib/python3.10/site-packages/charmcraft/application/commands/store.py", line 1681, in run
    libs_metadata = store.get_libraries_metadata_by_name(charm_libs)
  File "/snap/charmcraft/x1/lib/python3.10/site-packages/charmcraft/services/store.py", line 248, in get_libraries_metadata_by_name
    for lib in self.get_libraries_metadata(libraries)
  File "/snap/charmcraft/x1/lib/python3.10/site-packages/charmcraft/services/store.py", line 240, in get_libraries_metadata
    return self.anonymous_client.fetch_libraries_metadata(store_libs)
  File "/snap/charmcraft/x1/lib/python3.10/site-packages/charmcraft/store/client.py", line 103, in fetch_libraries_metadata
    response = self.request_urlpath_json("POST", "/v1/charm/libraries/bulk", json=libs)
  File "/snap/charmcraft/x1/lib/python3.10/site-packages/charmcraft/store/client.py", line 64, in request_urlpath_json
    response = self._http_client.request(method, self.api_base_url + urlpath, *args, **kwargs)
  File "/snap/charmcraft/x1/lib/python3.10/site-packages/craft_store/http_client.py", line 166, in request
    raise errors.StoreServerError(response)
craft_store.errors.StoreServerError: Store operation failed:
- None: Items need to include 'library_id' or 'package_id'
-----
```


Discussion:

Patch:
diff --git a/.github/workflows/spread-large.yaml b/.github/workflows/spread-large.yaml
index f1ef4f14..9f971ba1 100644
--- a/.github/workflows/spread-large.yaml
+++ b/.github/workflows/spread-large.yaml
@@ -6,31 +6,19 @@ on:
 
   workflow_dispatch:
 
+  # If we're editing this file, run it on the pull request.
+  pull_request:
+    paths:
+      - .github/workflows/spread-large.yaml
+
 # Only ever run one of this test at a time.
 concurrency:
   group: ${{ github.workflow }}
   cancel-in-progress: true
 
 jobs:
-  snap-build:
-    runs-on: ubuntu-22.04
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v4
-        with:
-          fetch-depth: 0
-      - name: Build snap
-        uses: snapcore/action-build@v1
-        id: charmcraft
-      - name: Upload snap artifact
-        uses: actions/upload-artifact@v4
-        with:
-          name: snap
-          path: ${{ steps.charmcraft.outputs.snap }}
-
-  pack-charm:
+  spread:
     runs-on: spread-installed
-    needs: [snap-build]
     strategy:
       fail-fast: false
       # Each of these tasks can spin up several machines in spread.
@@ -54,10 +42,9 @@ jobs:
         with:
           fetch-depth: 0
           submodules: true
-      - name: Download snap artifact
-        uses: actions/download-artifact@v4
-        with:
-          name: snap
+      - name: Download charmcraft snap
+        run: |
+          snap download --channel=latest/edge charmcraft
       - name: Spread k8s operator
         env:
           LAUNCHPAD_TOKEN: ${{ secrets.LAUNCHPAD_TOKEN }}
diff --git a/.github/workflows/spread.yaml b/.github/workflows/spread.yaml
index f1179c0d..0ff901b4 100644
--- a/.github/workflows/spread.yaml
+++ b/.github/workflows/spread.yaml
@@ -10,7 +10,7 @@ on:
 
 jobs:
   snap-build:
-    runs-on: ubuntu-22.04
+    runs-on: [self-hosted, amd64]
     steps:
       - name: Checkout code
         uses: actions/checkout@v4
diff --git a/.gitignore b/.gitignore
index 6866d3f6..3ac2a33d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -145,3 +145,6 @@ dmypy.json
 
 # Spread files
 .spread-reuse*.yaml
+
+# `snapcraft try` can be very useful
+/prime/
diff --git a/.readthedocs.yaml b/.readthedocs.yaml
index 59373ae9..189e51c2 100644
--- a/.readthedocs.yaml
+++ b/.readthedocs.yaml
@@ -11,8 +11,8 @@ build:
     python: "3.10"
   jobs:
     post_checkout:
-      - git fetch --tags --depth 1 # Also fetch tags
-      - git describe --always      # Useful for debugging
+      - git fetch --tags --depth 1  # Also fetch tags
+      - git describe --always       # Useful for debugging
 
 # Build documentation in the docs/ directory with Sphinx
 sphinx:
diff --git a/charmcraft/services/store.py b/charmcraft/services/store.py
index f5c56250..ce1ddd62 100644
--- a/charmcraft/services/store.py
+++ b/charmcraft/services/store.py
@@ -24,6 +24,7 @@ import craft_application
 import craft_store
 from craft_cli import emit
 from craft_store import models
+from craft_store.errors import StoreServerError
 from overrides import override
 
 from charmcraft import const, env, errors, store
@@ -244,7 +245,22 @@ class StoreService(BaseStoreService):
                 store_lib["patch"] = patch_version
             store_libs.append(store_lib)
 
-        return self.anonymous_client.fetch_libraries_metadata(store_libs)
+        try:
+            return self.anonymous_client.fetch_libraries_metadata(store_libs)
+        except StoreServerError as exc:
+            lib_names = [lib.lib for lib in libraries]
+            # Type ignore here because error_list is supposed to have string keys, but
+            # for whatever reason the store returns a null code for this one.
+            # https://bugs.launchpad.net/snapstore-server/+bug/1925065
+            if exc.error_list[None]["message"] == (  # type: ignore[index]
+                "Items need to include 'library_id' or 'package_id'"
+            ):
+                raise errors.LibraryError(
+                    "One or more declared charm-libs could not be found in the store.",
+                    details="Declared charm-libs: " + ", ".join(lib_names),
+                    resolution="Check the charm and library names in charmcraft.yaml",
+                ) from exc
+            raise
 
     def get_libraries_metadata_by_name(
         self, libraries: Sequence[CharmLib]
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 80597a40..0f6a93d6 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -12,7 +12,7 @@ craft-application==4.4.0
 craft-archives==2.0.1
 craft-cli==2.10.1
 craft-grammar==2.0.1
-craft-parts==2.1.3
+craft-parts==2.1.4
 craft-platforms==0.4.0
 craft-providers==2.0.4
 craft-store==3.0.2


Test patch:
diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml
index 6b4ad83c..5c80db8c 100644
--- a/.github/workflows/tests.yaml
+++ b/.github/workflows/tests.yaml
@@ -14,7 +14,7 @@ on:
 
 jobs:
   snap-build:
-    runs-on: ubuntu-22.04
+    runs-on: [self-hosted, amd64]
     steps:
       - name: Checkout code
         uses: actions/checkout@v4
diff --git a/tests/spread/dependencies/requirements/requirements-basic.txt b/tests/spread/dependencies/requirements/requirements-basic.txt
index bc96bbd0..87852332 100644
--- a/tests/spread/dependencies/requirements/requirements-basic.txt
+++ b/tests/spread/dependencies/requirements/requirements-basic.txt
@@ -1,3 +1,3 @@
 # Basic requirements.txt without hashes, but with various forms of reqs
-ops~=2.5
+ops~=2.15
 pyxdg<999.99
diff --git a/tests/spread/dependencies/requirements/requirements-hashes.txt b/tests/spread/dependencies/requirements/requirements-hashes.txt
index 1b017c46..e829c8ec 100644
--- a/tests/spread/dependencies/requirements/requirements-hashes.txt
+++ b/tests/spread/dependencies/requirements/requirements-hashes.txt
@@ -1,7 +1,7 @@
 # Requirements.txt file with hashes
-ops==2.6.0 \
-    --hash=sha256:387d5f5bd004b865f575ea4e25ca298bdca05b2b2beb2c51f203d5856c460cf6 \
-    --hash=sha256:b8ce352df4d17de483173dd2f8caf397e89585212d73b4d466b541b1ad3b8966
+ops==2.15.0 \
+    --hash=sha256:8e47ab8a814301776b0ff42b32544ebdece7f1639168d2c86dc7a25930d2e493 \
+    --hash=sha256:f3bad7417e98e8f390523fad097702eed16e99b38a25e9fe856aad226474b057
     # via -r requirements-basic.txt
 pyxdg==0.28 \
     --hash=sha256:3267bb3074e934df202af2ee0868575484108581e6f3cb006af1da35395e88b4 \
diff --git a/tests/unit/services/test_store.py b/tests/unit/services/test_store.py
index 55dadff4..f9776ffe 100644
--- a/tests/unit/services/test_store.py
+++ b/tests/unit/services/test_store.py
@@ -20,8 +20,10 @@ from typing import cast
 from unittest import mock
 
 import craft_store
+import craft_store.errors
 import distro
 import pytest
+import requests
 from craft_cli.pytest_plugin import RecordingEmitter
 from craft_store import models
 from hypothesis import given, strategies
@@ -34,12 +36,12 @@ from tests import get_fake_revision
 
 
 @pytest.fixture
-def store(service_factory) -> services.StoreService:
+def store(service_factory, mock_store_anonymous_client) -> services.StoreService:
     store = services.StoreService(
         app=application.APP_METADATA, services=service_factory
     )
     store.client = mock.Mock(spec_set=client.Client)
-    store.anonymous_client = mock.Mock(spec_set=client.AnonymousClient)
+    store.anonymous_client = mock_store_anonymous_client
     return store
 
 
@@ -305,3 +307,17 @@ def test_fetch_libraries_metadata(monkeypatch, store, libs, expected_call):
     store.anonymous_client.fetch_libraries_metadata.assert_called_once_with(
         expected_call
     )
+
+
+def test_get_libraries_metadata_name_error(
+    monkeypatch, store: services.StoreService, mock_store_anonymous_client: mock.Mock
+) -> None:
+    bad_response = requests.Response()
+    bad_response.status_code = 400
+    bad_response._content = b'{"error-list": [{"code": null, "message": "Items need to include \'library_id\' or \'package_id\'"}]}'
+    mock_store_anonymous_client.fetch_libraries_metadata.side_effect = (
+        craft_store.errors.StoreServerError(bad_response)
+    )
+
+    with pytest.raises(errors.LibraryError, match="One or more declared"):
+        store.get_libraries_metadata([CharmLib(lib="boop.snoot", version="-1")])