Repository: canonical/charmcraft

Issue:
`fetch-libs` crashes with an internal error
### Bug Description

After logging in and running `fetch-libs`, there is an internal error `charmcraft internal error: StoreServerError("Store operation failed:\n- None: Items need to include 'library_id' or 'package_id'")`

### To Reproduce

1. Create a multipass VM
2. Install `latest/edge`  of charmcraft
3. Include a `charm-libs` entry such as:

```yaml
charm-libs:
  - lib: observability_libs.juju_topology
    version: 0
```

4. run `fetch-libs`

### Environment

arm64 ubuntu jammy, charmcraft on `latest/edge`

### charmcraft.yaml

```shell
# This file configures Charmcraft.
# See https://juju.is/docs/sdk/charmcraft-config for guidance.

name: sample-flask

type: charm

bases:
  - build-on:
    - name: ubuntu
      channel: "22.04"
    run-on:
    - name: ubuntu
      channel: "22.04"

# (Required)
summary: A very short one-line summary of the flask application.

# (Required)
description: |
  A comprehensive overview of your Flask application.

extensions:
  - flask-framework

charm-libs:
  - lib: observability_libs.juju_topology
    version: 0

# Uncomment the integrations used by your application
# requires:
#   mysql:
#     interface: mysql_client
#     limit: 1
#   postgresql:
#     interface: postgresql_client
#     limit: 1
```


### Relevant log output

```shell
cat /home/ubuntu/.local/state/charmcraft/log/charmcraft-20240509-174417.997582.log
2024-05-09 17:44:17.998 Starting charmcraft, version 3.0.0.post39+ge2ad204
2024-05-09 17:44:17.998 Log verbosity level set to BRIEF
2024-05-09 17:44:17.998 Configuring application...
2024-05-09 17:44:17.998 Preparing application...
2024-05-09 17:44:17.998 Build plan: platform=None, build_for=None
2024-05-09 17:44:17.998 Loading project file '/home/ubuntu/src/sample-flask/charm/charmcraft.yaml'
2024-05-09 17:44:18.000 Processing grammar (on arm64 for arm64)
2024-05-09 17:44:18.000 Processing grammar for plugin: charm
2024-05-09 17:44:18.000 Processing grammar for source: .
2024-05-09 17:44:18.000 Running charmcraft fetch-libs on host
2024-05-09 17:44:18.000 Setting up StoreService
2024-05-09 17:44:18.074 Falling back to file based storage
2024-05-09 17:44:18.074 Getting library metadata from charmhub
2024-05-09 17:44:18.074 Retrieving credentials for 'charmcraft' on 'api.charmhub.io' from keyring 'auth FileKeyring'.
2024-05-09 17:44:18.074 HTTP 'POST' for 'https://api.charmhub.io/v1/charm/libraries/bulk' with params None and headers {'Authorization': '<macaroon>', 'User-Agent': 'Charmcraft/3.0.0.post39+ge2ad204 (Linux 5.15.0-106-generic; aarch64; CPython 3.10.12; Ubuntu 22.04)'}
2024-05-09 17:44:18.258 charmcraft internal error: StoreServerError("Store operation failed:\n- None: Items need to include 'library_id' or 'package_id'")
2024-05-09 17:44:18.265 Traceback (most recent call last):
2024-05-09 17:44:18.265   File "/snap/charmcraft/3739/lib/python3.10/site-packages/craft_application/application.py", line 517, in run
2024-05-09 17:44:18.265     return_code = dispatcher.run() or 0
2024-05-09 17:44:18.265   File "/snap/charmcraft/3739/lib/python3.10/site-packages/craft_cli/dispatcher.py", line 487, in run
2024-05-09 17:44:18.265     return self._loaded_command.run(self._parsed_command_args)
2024-05-09 17:44:18.265   File "/snap/charmcraft/3739/lib/python3.10/site-packages/charmcraft/application/commands/store.py", line 1679, in run
2024-05-09 17:44:18.265     libs_metadata = store.get_libraries_metadata_by_name(charm_libs)
2024-05-09 17:44:18.265   File "/snap/charmcraft/3739/lib/python3.10/site-packages/charmcraft/services/store.py", line 220, in get_libraries_metadata_by_name
2024-05-09 17:44:18.265     for lib in self.get_libraries_metadata(libraries)
2024-05-09 17:44:18.265   File "/snap/charmcraft/3739/lib/python3.10/site-packages/charmcraft/services/store.py", line 212, in get_libraries_metadata
2024-05-09 17:44:18.265     return self.client.fetch_libraries_metadata(store_libs)
2024-05-09 17:44:18.265   File "/snap/charmcraft/3739/lib/python3.10/site-packages/charmcraft/store/client.py", line 202, in fetch_libraries_metadata
2024-05-09 17:44:18.265     response = self.request_urlpath_json("POST", "/v1/charm/libraries/bulk", json=libs)
2024-05-09 17:44:18.265   File "/snap/charmcraft/3739/lib/python3.10/site-packages/charmcraft/store/client.py", line 134, in request_urlpath_json
2024-05-09 17:44:18.265     response = super().request(method, self.api_base_url + urlpath, *args, **kwargs)
2024-05-09 17:44:18.265   File "/snap/charmcraft/3739/lib/python3.10/site-packages/craft_store/base_client.py", line 186, in request
2024-05-09 17:44:18.265     return self.http_client.request(
2024-05-09 17:44:18.265   File "/snap/charmcraft/3739/lib/python3.10/site-packages/craft_store/http_client.py", line 166, in request
2024-05-09 17:44:18.265     raise errors.StoreServerError(response)
2024-05-09 17:44:18.265 craft_store.errors.StoreServerError: Store operation failed:
2024-05-09 17:44:18.265 - None: Items need to include 'library_id' or 'package_id'
2024-05-09 17:44:18.265 Full execution log: '/home/ubuntu/.local/state/charmcraft/log/charmcraft-20240509-174417.997582.log'
```


Discussion:

Patch:
diff --git a/charmcraft/application/commands/store.py b/charmcraft/application/commands/store.py
index 0e2374bb..a0df916d 100644
--- a/charmcraft/application/commands/store.py
+++ b/charmcraft/application/commands/store.py
@@ -1700,10 +1700,11 @@ class FetchLibs(CharmcraftCommand):
             lib_name = f"{lib_md.charm_name}.{lib_md.lib_name}"
             local_lib = local_libs.get(lib_name)
             if local_lib and local_lib.content_hash == lib_md.content_hash:
-                emit.debug(
+                emit.progress(
                     f"Skipping {lib_name} because the same file already exists on "
                     f"disk (hash: {lib_md.content_hash}). "
-                    "Delete the file and re-run 'charmcraft fetch-libs' to force re-download."
+                    "Delete the file and re-run 'charmcraft fetch-libs' to force re-download.",
+                    permanent=True,
                 )
                 continue
             lib_name = utils.get_lib_module_name(lib_md.charm_name, lib_md.lib_name, lib_md.api)
@@ -1722,6 +1723,7 @@ class FetchLibs(CharmcraftCommand):
             lib_path = utils.get_lib_path(lib_md.charm_name, lib_md.lib_name, lib_md.api)
             lib_path.parent.mkdir(exist_ok=True, parents=True)
             lib_path.write_text(lib.content)
+            emit.debug(f"Downloaded {lib_name}.")
 
         emit.message(f"Downloaded {downloaded_libs} charm libraries.")
 
diff --git a/charmcraft/application/main.py b/charmcraft/application/main.py
index bec78bb3..1768cfcb 100644
--- a/charmcraft/application/main.py
+++ b/charmcraft/application/main.py
@@ -52,6 +52,7 @@ class Charmcraft(Application):
         super().__init__(app=app, services=services)
         self._global_args: dict[str, Any] = {}
         self._dispatcher: craft_cli.Dispatcher | None = None
+        self._cli_loggers |= {"charmcraft"}
 
     @property
     def command_groups(self) -> list[craft_cli.CommandGroup]:
diff --git a/charmcraft/models/project.py b/charmcraft/models/project.py
index cc32425e..621b7dd7 100644
--- a/charmcraft/models/project.py
+++ b/charmcraft/models/project.py
@@ -125,8 +125,8 @@ class CharmLib(models.CraftBaseModel):
     """A Charm library dependency for this charm."""
 
     lib: str = pydantic.Field(
-        title="Library Path (e.g. my_charm.my_library)",
-        regex=r"[a-z0-9_]+\.[a-z0-9_]+",
+        title="Library Path (e.g. my-charm.my_library)",
+        regex=r"[a-z][a-z0-9_-]+\.[a-z][a-z0-9_]+",
     )
     version: str = pydantic.Field(
         title="Version filter for the charm. Either an API version or a specific [api].[patch].",
@@ -141,17 +141,18 @@ class CharmLib(models.CraftBaseModel):
             raise ValueError(
                 f"Library name invalid. Expected '[charm_name].[lib_name]', got {value!r}"
             )
-        if not re.fullmatch("[a-z0-9_]+", charm_name):
-            if "-" in charm_name:
-                raise ValueError(
-                    f"Invalid charm name in lib {value!r}. Try replacing hyphens ('-') with underscores ('_')."
-                )
+        # Accept python-importable charm names, but convert them to store-accepted names.
+        if "_" in charm_name:
+            charm_name = charm_name.replace("_", "-")
+        if not re.fullmatch("[a-z0-9_-]+", charm_name):
             raise ValueError(
                 f"Invalid charm name for lib {value!r}. Value {charm_name!r} is invalid."
             )
         if not re.fullmatch("[a-z0-9_]+", lib_name):
-            raise ValueError(f"Library name {lib_name!r} is invalid.")
-        return str(value)
+            raise ValueError(
+                f"Library name {lib_name!r} is invalid. Library names must be valid Python module names."
+            )
+        return f"{charm_name}.{lib_name}"
 
     @pydantic.validator("version", pre=True)
     def _validate_api_version(cls, value: str) -> str:
@@ -640,11 +641,11 @@ class PlatformCharm(CharmcraftProject):
     summary: CharmcraftSummaryStr
     description: str
 
-    base: BaseStr
+    # Silencing pyright because it complains about missing default value
+    base: BaseStr  # pyright: ignore[reportGeneralTypeIssues]
     build_base: BuildBaseStr | None = None
-    platforms: dict[str, Platform | None]
-
-    parts: dict[str, dict[str, Any]]  # craft-parts parts
+    platforms: dict[str, Platform | None]  # pyright: ignore[reportGeneralTypeIssues]
+    parts: dict[str, dict[str, Any]]  # pyright: ignore[reportGeneralTypeIssues]
 
     actions: dict[str, Any] | None
     assumes: list[str | dict[str, list | dict]] | None
diff --git a/charmcraft/store/client.py b/charmcraft/store/client.py
index 54a0915d..9c013260 100644
--- a/charmcraft/store/client.py
+++ b/charmcraft/store/client.py
@@ -199,9 +199,14 @@ class Client(craft_store.StoreClient):
 
         http://api.charmhub.io/docs/libraries.html#fetch_libraries
         """
+        emit.trace(
+            f"Fetching library metadata from charmhub: {libs}",
+        )
         response = self.request_urlpath_json("POST", "/v1/charm/libraries/bulk", json=libs)
         if "libraries" not in response:
             raise CraftError(
                 "Server returned invalid response while querying libraries", details=str(response)
             )
-        return [Library.from_dict(lib) for lib in response["libraries"]]
+        converted_response = [Library.from_dict(lib) for lib in response["libraries"]]
+        emit.trace(f"Store response: {converted_response}")
+        return converted_response
diff --git a/charmcraft/templates/init-kubernetes/tox.ini.j2 b/charmcraft/templates/init-kubernetes/tox.ini.j2
index a7be90a1..1b06be98 100644
--- a/charmcraft/templates/init-kubernetes/tox.ini.j2
+++ b/charmcraft/templates/init-kubernetes/tox.ini.j2
@@ -29,7 +29,7 @@ deps =
     ruff
 commands =
     ruff format {[vars]all_path}
-    ruff --fix {[vars]all_path}
+    ruff check --fix {[vars]all_path}
 
 [testenv:lint]
 description = Check code against coding style standards
@@ -41,7 +41,7 @@ commands =
     # and uncomment the following line
     # codespell {[vars]lib_path}
     codespell {tox_root}
-    ruff {[vars]all_path}
+    ruff check {[vars]all_path}
     ruff format --check --diff {[vars]all_path}
 
 [testenv:unit]
diff --git a/charmcraft/templates/init-machine/tox.ini.j2 b/charmcraft/templates/init-machine/tox.ini.j2
index a7be90a1..1b06be98 100644
--- a/charmcraft/templates/init-machine/tox.ini.j2
+++ b/charmcraft/templates/init-machine/tox.ini.j2
@@ -29,7 +29,7 @@ deps =
     ruff
 commands =
     ruff format {[vars]all_path}
-    ruff --fix {[vars]all_path}
+    ruff check --fix {[vars]all_path}
 
 [testenv:lint]
 description = Check code against coding style standards
@@ -41,7 +41,7 @@ commands =
     # and uncomment the following line
     # codespell {[vars]lib_path}
     codespell {tox_root}
-    ruff {[vars]all_path}
+    ruff check {[vars]all_path}
     ruff format --check --diff {[vars]all_path}
 
 [testenv:unit]
diff --git a/charmcraft/templates/init-simple/tox.ini.j2 b/charmcraft/templates/init-simple/tox.ini.j2
index a7be90a1..1b06be98 100644
--- a/charmcraft/templates/init-simple/tox.ini.j2
+++ b/charmcraft/templates/init-simple/tox.ini.j2
@@ -29,7 +29,7 @@ deps =
     ruff
 commands =
     ruff format {[vars]all_path}
-    ruff --fix {[vars]all_path}
+    ruff check --fix {[vars]all_path}
 
 [testenv:lint]
 description = Check code against coding style standards
@@ -41,7 +41,7 @@ commands =
     # and uncomment the following line
     # codespell {[vars]lib_path}
     codespell {tox_root}
-    ruff {[vars]all_path}
+    ruff check {[vars]all_path}
     ruff format --check --diff {[vars]all_path}
 
 [testenv:unit]
diff --git a/pyproject.toml b/pyproject.toml
index e6c2bb2a..086ba5b5 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -64,7 +64,7 @@ lint = [
 ]
 types = [
     "mypy[reports]~=1.5",
-    "pyright==1.1.354",
+    "pyright==1.1.366",
     "types-python-dateutil",
     "types-requests<2.31.0.20240312",  # Frozen until we can get urllib3 v2
     "types-setuptools",


Test patch:
diff --git a/tests/unit/models/test_project.py b/tests/unit/models/test_project.py
index f4c2cc45..67046112 100644
--- a/tests/unit/models/test_project.py
+++ b/tests/unit/models/test_project.py
@@ -193,27 +193,33 @@ def test_build_info_from_build_on_run_on_basic(
 
 
 @pytest.mark.parametrize(
-    "lib_name",
-    ["charm.lib", "charm_with_hyphens.lib", "charm.lib_with_hyphens", "charm0.number_0_lib"],
+    ("lib_name", "expected_lib_name"),
+    [
+        ("charm.lib", "charm.lib"),
+        ("charm_with_underscores.lib", "charm-with-underscores.lib"),
+        ("charm-with-hyphens.lib", "charm-with-hyphens.lib"),
+        ("charm.lib_with_hyphens", "charm.lib_with_hyphens"),
+        ("charm0.number_0_lib", "charm0.number_0_lib"),
+    ],
 )
 @pytest.mark.parametrize("lib_version", ["0", "1", "2.0", "2.1", "3.14"])
-def test_create_valid_charm_lib(lib_name, lib_version):
-    project.CharmLib.unmarshal({"lib": lib_name, "version": lib_version})
+def test_create_valid_charm_lib(lib_name: str, expected_lib_name: str, lib_version: str):
+    lib = project.CharmLib.unmarshal({"lib": lib_name, "version": lib_version})
+    assert lib.lib == expected_lib_name
 
 
 @pytest.mark.parametrize(
     ("name", "error_match"),
     [
         ("boop", r"Library name invalid. Expected '\[charm_name\].\[lib_name\]', got 'boop'"),
-        (
-            "raw-charm-name.valid_lib",
-            r"Invalid charm name in lib 'raw-charm-name.valid_lib'. Try replacing hyphens \('-'\) with underscores \('_'\).",
-        ),
         (
             "Invalid charm name.valid_lib",
             "Invalid charm name for lib 'Invalid charm name.valid_lib'. Value 'Invalid charm name' is invalid",
         ),
-        ("my_charm.invalid library name", "Library name 'invalid library name' is invalid."),
+        (
+            "my_charm.invalid-library-name",
+            "Library name 'invalid-library-name' is invalid. Library names must be valid Python module names.",
+        ),
     ],
 )
 def test_invalid_charm_lib_name(name: str, error_match: str):
diff --git a/tests/unit/models/valid_charms_yaml/full-bases.yaml b/tests/unit/models/valid_charms_yaml/full-bases.yaml
index 170fafd8..d239bde6 100644
--- a/tests/unit/models/valid_charms_yaml/full-bases.yaml
+++ b/tests/unit/models/valid_charms_yaml/full-bases.yaml
@@ -25,9 +25,9 @@ parts:
   another-part:
     plugin: nil
 charm-libs:
-  - lib: my_charm.my_lib
+  - lib: my-charm.my_lib
     version: '1'
-  - lib: other_charm.lib
+  - lib: other-charm.lib
     version: '0.123'
 bases:
 - build-on:
diff --git a/tests/unit/models/valid_charms_yaml/full-platforms.yaml b/tests/unit/models/valid_charms_yaml/full-platforms.yaml
index 855e4456..93104182 100644
--- a/tests/unit/models/valid_charms_yaml/full-platforms.yaml
+++ b/tests/unit/models/valid_charms_yaml/full-platforms.yaml
@@ -25,9 +25,9 @@ parts:
   another-part:
     plugin: nil
 charm-libs:
-  - lib: my_charm.my_lib
+  - lib: my-charm.my_lib
     version: '1'
-  - lib: other_charm.lib
+  - lib: other-charm.lib
     version: '0.123'
 base: ubuntu@24.04
 build-base: ubuntu@devel