Repository: fwupd/fwupd (C)

Issue:
fwupdmgr get-devices --json gives file name in URI
Getting `"Uri" : "b1275193349c3ebd1db3821462143ac859302afe3572060fab7ee4793b4892bd-DellDockFirmwareUpdateLinux_01.00.16.cab"` when I did `fwupdmgr get-devices --json` for all releases. This should ideally be the relative URI, based on the MetadataURI in lvfs.conf


Discussion: > Uri....

You mean Locations? or is this 1.9.x? e.g. I get: 

    {
      "Name" : "WDC PC SN720 SDAQNTW-256G-1001",
      "DeviceId" : "08e1798bf5d9cb56a0290b552cab6c1a371b5089",
      "InstanceIds" : [
        "NVME\\VEN_15B7&DEV_5002",
        "NVME\\VEN_15B7&DEV_5002&SUBSYS_15B75002",
        "WDC PC SN720 SDAQNTW-256G-1001"
      ],
      "Guid" : [
        "ff2112dc-038c-596d-90ca-d43c5077c6ec",
        "c528df4b-7972-5880-8cb1-330415e2dc6a",
        "237776ee-0bcd-5fe9-8dc8-6984a2d36ba0"
      ],
      "Serial" : "183985804591",
      "Summary" : "NVM Express solid state drive",
      "Plugin" : "nvme",
      "Protocols" : [
        "org.nvmexpress"
      ],
      "Flags" : [
        "internal",
        "updatable",
        "require-ac",
        "supported",
        "needs-reboot",
        "usable-during-update",
        "can-emulation-tag"
      ],
      "Vendor" : "Sandisk Corp",
      "VendorIds" : [
        "PCI:0x15B7"
      ],
      "Version" : "10190101",
      "VersionFormat" : "plain",
      "Icons" : [
        "drive-harddisk"
      ],
      "Created" : 1732129505,
      "Releases" : [
        {
          "AppstreamId" : "com.lenovo.PCSN720.firmware",
          "ReleaseId" : "4305",
          "RemoteId" : "lvfs",
          "Name" : "PCSN720",
          "Summary" : "WD PCSN720 Firmware Update for Lenovo PC",
          "Description" : "<p>Supported devices and firmware version: WDC PCSN720 NVMe SSD Firmware for Lenovo PC</p>",
          "Version" : "10190101",
          "Filename" : "Lenovo-WD-PCSN720-NVMe-10190101.cab",
          "Protocol" : "org.nvmexpress",
          "Categories" : [
            "X-SolidStateDrive",
            "X-Device"
          ],
          "Checksum" : [
            "aed65b1eae17a6b8dbe1441bad4c6f99fbe17c50",
            "8e2a52b74fcdbcf2f82bf2e0f6b750d6f098be887bb1e4c6acba3b63454a4f73"
          ],
          "License" : "LicenseRef-proprietary",
          "Size" : 2277376,
          "Created" : 1577426837,
          "Locations" : [
            "https://fwupd.org/downloads/8e2a52b74fcdbcf2f82bf2e0f6b750d6f098be887bb1e4c6acba3b63454a4f73-Lenovo-WD-PCSN720-NVMe-10190101.cab"
          ],
          "Homepage" : "https://www.lenovo.com",
          "Vendor" : "Lenovo",
          "Flags" : [
            "trusted-metadata"
          ]
        }
      ]
    }

I have both:
```
"Locations" : [
            "b1275193349c3ebd1db3821462143ac859302afe3572060fab7ee4793b4892bd-DellDockFirmwareUpdateLinux_01.00.16.cab"
          ],
          "Uri" : "b1275193349c3ebd1db3821462143ac859302afe3572060fab7ee4793b4892bd-DellDockFirmwareUpdateLinux_01.00.16.cab",

```
Yes, this is 1.9.22
Hmm, I get a full URL for 1_9_X too:

          "Locations" : [
            "https://cdn.fwupd.org/downloads/8e2a52b74fcdbcf2f82bf2e0f6b750d6f098be887bb1e4c6acba3b63454a4f73-Lenovo-WD-PCSN720-NVMe-10190101.cab"
          ],
          "Uri" : "https://cdn.fwupd.org/downloads/8e2a52b74fcdbcf2f82bf2e0f6b750d6f098be887bb1e4c6acba3b63454a4f73-Lenovo-WD-PCSN720-NVMe-10190101.cab",

One thing it might be; are you still using the gzip metadata?
Nope, I tried with xz and gz, and those both worked. Two things;

 * can you paste the meson configure log, i.e. the line where it has `lvfs_metadata_format`
 * can you paste your lvfs.conf please

It's even correct using the 1.9.22 tag. Something is weird! :)
I think the lvfs conf in use must be missing `FirmwareBaseUri`
lvfs.conf:
```
[fwupd Remote]

# this remote provides metadata and firmware marked as 'stable' from the LVFS
Enabled=true
Title=Linux Vendor Firmware Service
MetadataURI=https://storage.googleapis.com/chromeos-localmirror/lvfs/firmware.xml.xz
ReportURI=https://fwupd.org/lvfs/firmware/report
SecurityReportURI=https://fwupd.org/lvfs/hsireports/upload
AutomaticReports=false
AutomaticSecurityReports=false
ApprovalRequired=false
FirmwareBaseURI=https://storage.googleapis.com/chromeos-localmirror/lvfs/

```

Correction: it is v1.9.21.
Working on getting meson config

> I think the lvfs conf in use must be missing `FirmwareBaseUri`

Oh, based on what is shared - other way around.  It's setting `FirmwareBaseUri`.  We don't normally do that and so it goes down  a different path than we normally do in fwupd-remote.c in `fwupd_remote_build_uri`
We dropped `FirmwareBaseUri` in main (see 861c9804a35897eda04b5d6a4918a8ca46d2f27f).  Are you guys moving up to 2.0.2 after the next release?  If so, this might not be worth changing anything in 1.9.x..
Removing the line `FirmwareBaseUri` fixed the issue, but based on this [pull request](https://github.com/fwupd/fwupd/pull/6877#issuecomment-1980426999), the old way should also work? Don't want older builds to start failing even after we uprev to 2.0.2
Confirming that 2.0.2 works even with the `FirmwareBaseUri` line. 1.9.22 breaks.
The pull request was purely to 2.0.x, not planned to backport.  So it sounds like if you move up to 2.0.2 things will be fine, and 1.9.x has this bug.  If your next release is 1.9.27 we should fix in 1.9.x, if it's 2.0.2 then I think this is "close won't fix"
Our next release is 2.0.1 but we still have all our devices on 1.9.22 until we release 2.0.2 to stable (~3 months out). Not sure what changed to have suddenly caused this but until a couple of days ago 1.9.22 was working, now suddenly the URIs are showing only filenames. Maybe something changed on lvfs side?
2.0.1 should work too.  Can you have a try with that?
I've reverted the LVFS "fix" until we understand what's happened. It's also broken one of the ODMs too.
So, I dug into this a bit today. The LVFS change that made the URL into a filename was probably bogus; not because of mirroring -- but more that we started using the CDN to distribute packages.

e.g. `https://fwupd.org/downloads/fc3feb015df2710fcfa07583d31b5975ee398357016699cfff067f422ab91e13-DBXUpdate-20230509-x64.cab` -> `https://cdn.fwupd.org/downloads/fc3feb015df2710fcfa07583d31b5975ee398357016699cfff067f422ab91e13-DBXUpdate-20230509-x64.cab`

i.e. I think we actually do want a different URL for firmware binaries to the metadata. The alternative is doing something like `s/cdn./` but that feels like a bit of a hack. @superm1 would you be against me bringing back FirmwareBaseURI for 2.0.2?
> .e. I think we actually do want a different URL for firmware binaries to the metadata. The alternative is doing something like s/cdn./ but that feels like a bit of a hack. @superm1 would you be against me bringing back FirmwareBaseURI for 2.0.2?

That's fine



Fix (patch):
diff --git a/libfwupd/fwupd-remote-private.h b/libfwupd/fwupd-remote-private.h
index 263527c5fdc1..5d0c0447df63 100644
--- a/libfwupd/fwupd-remote-private.h
+++ b/libfwupd/fwupd-remote-private.h
@@ -49,6 +49,9 @@ void
 fwupd_remote_set_password(FwupdRemote *self, const gchar *password) G_GNUC_NON_NULL(1);
 void
 fwupd_remote_set_report_uri(FwupdRemote *self, const gchar *report_uri) G_GNUC_NON_NULL(1);
+void
+fwupd_remote_set_firmware_base_uri(FwupdRemote *self, const gchar *firmware_base_uri)
+    G_GNUC_NON_NULL(1);
 
 void
 fwupd_remote_set_remotes_dir(FwupdRemote *self, const gchar *directory) G_GNUC_NON_NULL(1);
diff --git a/libfwupd/fwupd-remote.c b/libfwupd/fwupd-remote.c
index 2d5d28aab1de..5cc99382159b 100644
--- a/libfwupd/fwupd-remote.c
+++ b/libfwupd/fwupd-remote.c
@@ -157,6 +157,7 @@ fwupd_remote_add_json(FwupdCodec *codec, JsonBuilder *builder, FwupdCodecFlags f
 	fwupd_codec_json_append(builder, "ReportUri", priv->report_uri);
 	fwupd_codec_json_append(builder, "MetadataUri", priv->metadata_uri);
 	fwupd_codec_json_append(builder, "MetadataUriSig", priv->metadata_uri_sig);
+	fwupd_codec_json_append(builder, "FirmwareBaseUri", priv->firmware_base_uri);
 	fwupd_codec_json_append(builder, "Username", priv->username);
 	fwupd_codec_json_append(builder, "Password", priv->password);
 	fwupd_codec_json_append(builder, "Title", priv->title);
@@ -510,7 +511,10 @@ fwupd_remote_set_filename_source(FwupdRemote *self, const gchar *filename_source
 }
 
 static gchar *
-fwupd_remote_build_uri(FwupdRemote *self, const gchar *url_noauth, GError **error)
+fwupd_remote_build_uri(FwupdRemote *self,
+		       const gchar *base_uri,
+		       const gchar *url_noauth,
+		       GError **error)
 {
 	FwupdRemotePrivate *priv = GET_PRIVATE(self);
 #ifdef HAVE_LIBCURL
@@ -532,8 +536,27 @@ fwupd_remote_build_uri(FwupdRemote *self, const gchar *url_noauth, GError **erro
 		url = g_strdup(url_noauth);
 	}
 
-	/* use the base URI of the metadata to build the full path */
-	if (g_strstr_len(url, -1, "/") == NULL) {
+	/* create URI, substituting if required */
+	if (base_uri != NULL) {
+		g_autofree gchar *basename = NULL;
+		g_autofree gchar *path_new = NULL;
+		g_autoptr(curlptr) path = NULL;
+		g_autoptr(CURLU) uri_tmp = curl_url();
+		if (curl_url_set(uri_tmp, CURLUPART_URL, url, 0) != CURLUE_OK) {
+			g_set_error(error,
+				    FWUPD_ERROR,
+				    FWUPD_ERROR_INVALID_FILE,
+				    "Failed to parse url '%s'",
+				    url);
+			return NULL;
+		}
+		(void)curl_url_get(uri_tmp, CURLUPART_PATH, &path, 0);
+		basename = g_path_get_basename(path);
+		path_new = g_build_filename(priv->firmware_base_uri, basename, NULL);
+		(void)curl_url_set(uri, CURLUPART_URL, path_new, 0);
+
+		/* use the base URI of the metadata to build the full path */
+	} else if (g_strstr_len(url, -1, "/") == NULL) {
 		g_autofree gchar *basename = NULL;
 		g_autofree gchar *path_new = NULL;
 		g_autoptr(curlptr) path = NULL;
@@ -574,6 +597,10 @@ fwupd_remote_build_uri(FwupdRemote *self, const gchar *url_noauth, GError **erro
 	(void)curl_url_get(uri, CURLUPART_URL, &tmp_uri, 0);
 	return g_strdup(tmp_uri);
 #else
+	if (priv->firmware_base_uri != NULL) {
+		g_autofree gchar *basename = g_path_get_basename(url_noauth);
+		return g_build_filename(priv->firmware_base_uri, basename, NULL);
+	}
 	if (g_strstr_len(url_noauth, -1, "/") == NULL) {
 		g_autofree gchar *basename = g_path_get_dirname(priv->metadata_uri);
 		return g_build_filename(basename, url_noauth, NULL);
@@ -613,6 +640,30 @@ fwupd_remote_set_metadata_uri(FwupdRemote *self, const gchar *metadata_uri)
 	priv->metadata_uri_sig = g_strconcat(metadata_uri, ".jcat", NULL);
 }
 
+/**
+ * fwupd_remote_set_firmware_base_uri:
+ * @self: a #FwupdRemote
+ * @firmware_base_uri: (nullable): base URI for firmware
+ *
+ * Sets the firmware base URI.
+ *
+ * NOTE: This has to be set after MetadataURI.
+ *
+ * Since: 2.0.2
+ **/
+void
+fwupd_remote_set_firmware_base_uri(FwupdRemote *self, const gchar *firmware_base_uri)
+{
+	FwupdRemotePrivate *priv = GET_PRIVATE(self);
+
+	/* not changed */
+	if (g_strcmp0(priv->firmware_base_uri, firmware_base_uri) == 0)
+		return;
+
+	g_free(priv->firmware_base_uri);
+	priv->firmware_base_uri = g_strdup(firmware_base_uri);
+}
+
 /**
  * fwupd_remote_set_report_uri:
  * @self: a #FwupdRemote
@@ -1238,10 +1289,11 @@ fwupd_remote_get_checksum_metadata(FwupdRemote *self)
 gchar *
 fwupd_remote_build_firmware_uri(FwupdRemote *self, const gchar *url, GError **error)
 {
+	FwupdRemotePrivate *priv = GET_PRIVATE(self);
 	g_return_val_if_fail(FWUPD_IS_REMOTE(self), NULL);
 	g_return_val_if_fail(url != NULL, NULL);
 	g_return_val_if_fail(error == NULL || *error == NULL, NULL);
-	return fwupd_remote_build_uri(self, url, error);
+	return fwupd_remote_build_uri(self, priv->firmware_base_uri, url, error);
 }
 
 /**
@@ -1261,7 +1313,7 @@ fwupd_remote_build_report_uri(FwupdRemote *self, GError **error)
 	FwupdRemotePrivate *priv = GET_PRIVATE(self);
 	g_return_val_if_fail(FWUPD_IS_REMOTE(self), NULL);
 	g_return_val_if_fail(error == NULL || *error == NULL, NULL);
-	return fwupd_remote_build_uri(self, priv->report_uri, error);
+	return fwupd_remote_build_uri(self, NULL, priv->report_uri, error);
 }
 
 /**
@@ -1281,7 +1333,7 @@ fwupd_remote_build_metadata_sig_uri(FwupdRemote *self, GError **error)
 	FwupdRemotePrivate *priv = GET_PRIVATE(self);
 	g_return_val_if_fail(FWUPD_IS_REMOTE(self), NULL);
 	g_return_val_if_fail(error == NULL || *error == NULL, NULL);
-	return fwupd_remote_build_uri(self, priv->metadata_uri_sig, error);
+	return fwupd_remote_build_uri(self, NULL, priv->metadata_uri_sig, error);
 }
 
 /**
@@ -1301,7 +1353,7 @@ fwupd_remote_build_metadata_uri(FwupdRemote *self, GError **error)
 	FwupdRemotePrivate *priv = GET_PRIVATE(self);
 	g_return_val_if_fail(FWUPD_IS_REMOTE(self), NULL);
 	g_return_val_if_fail(error == NULL || *error == NULL, NULL);
-	return fwupd_remote_build_uri(self, priv->metadata_uri, error);
+	return fwupd_remote_build_uri(self, NULL, priv->metadata_uri, error);
 }
 
 /**
@@ -1468,6 +1520,24 @@ fwupd_remote_get_metadata_uri_sig(FwupdRemote *self)
 	return priv->metadata_uri_sig;
 }
 
+/**
+ * fwupd_remote_get_firmware_base_uri:
+ * @self: a #FwupdRemote
+ *
+ * Gets the base URI for firmware.
+ *
+ * Returns: (transfer none): a URI, or %NULL for unset.
+ *
+ * Since: 2.0.2
+ **/
+const gchar *
+fwupd_remote_get_firmware_base_uri(FwupdRemote *self)
+{
+	FwupdRemotePrivate *priv = GET_PRIVATE(self);
+	g_return_val_if_fail(FWUPD_IS_REMOTE(self), NULL);
+	return priv->firmware_base_uri;
+}
+
 /**
  * fwupd_remote_needs_refresh:
  * @self: a #FwupdRemote
@@ -1563,6 +1633,8 @@ fwupd_remote_from_variant_iter(FwupdCodec *codec, GVariantIter *iter)
 			priv->mtime = g_variant_get_uint64(value);
 		} else if (g_strcmp0(key, "RefreshInterval") == 0) {
 			priv->refresh_interval = g_variant_get_uint64(value);
+		} else if (g_strcmp0(key, "FirmwareBaseUri") == 0) {
+			fwupd_remote_set_firmware_base_uri(self, g_variant_get_string(value, NULL));
 		} else if (g_strcmp0(key, "AutomaticReports") == 0) {
 			/* we can probably stop doing proxying flags when we next branch */
 			if (g_variant_get_boolean(value))
@@ -1639,6 +1711,12 @@ fwupd_remote_add_variant(FwupdCodec *codec, GVariantBuilder *builder, FwupdCodec
 				      "ReportUri",
 				      g_variant_new_string(priv->report_uri));
 	}
+	if (priv->firmware_base_uri != NULL) {
+		g_variant_builder_add(builder,
+				      "{sv}",
+				      "FirmwareBaseUri",
+				      g_variant_new_string(priv->firmware_base_uri));
+	}
 	if (priv->priority != 0) {
 		g_variant_builder_add(builder,
 				      "{sv}",
diff --git a/libfwupd/fwupd-remote.h b/libfwupd/fwupd-remote.h
index c9eb23cfc36b..133e6afcbc22 100644
--- a/libfwupd/fwupd-remote.h
+++ b/libfwupd/fwupd-remote.h
@@ -163,6 +163,8 @@ fwupd_remote_get_filename_cache_sig(FwupdRemote *self) G_GNUC_NON_NULL(1);
 const gchar *
 fwupd_remote_get_filename_source(FwupdRemote *self) G_GNUC_NON_NULL(1);
 const gchar *
+fwupd_remote_get_firmware_base_uri(FwupdRemote *self) G_GNUC_NON_NULL(1);
+const gchar *
 fwupd_remote_get_report_uri(FwupdRemote *self) G_GNUC_NON_NULL(1);
 const gchar *
 fwupd_remote_get_metadata_uri(FwupdRemote *self) G_GNUC_NON_NULL(1);
diff --git a/libfwupd/fwupd.map b/libfwupd/fwupd.map
index 9f8c14a94ba0..e89f13fb6d8a 100644
--- a/libfwupd/fwupd.map
+++ b/libfwupd/fwupd.map
@@ -993,3 +993,10 @@ LIBFWUPD_2.0.1 {
     fwupd_client_get_details_finish;
   local: *;
 } LIBFWUPD_2.0.0;
+
+LIBFWUPD_2.0.2 {
+  global:
+    fwupd_remote_get_firmware_base_uri;
+    fwupd_remote_set_firmware_base_uri;
+  local: *;
+} LIBFWUPD_2.0.1;
diff --git a/src/fu-engine.c b/src/fu-engine.c
index d27930b28a4d..3c62b44a3fa4 100644
--- a/src/fu-engine.c
+++ b/src/fu-engine.c
@@ -918,6 +918,7 @@ fu_engine_modify_remote(FuEngine *self,
 	    "AutomaticReports",
 	    "AutomaticSecurityReports",
 	    "Enabled",
+	    "FirmwareBaseURI",
 	    "MetadataURI",
 	    "ReportURI",
 	    "Username",
diff --git a/src/fu-remote.c b/src/fu-remote.c
index 570c9fd42452..256122b924fa 100644
--- a/src/fu-remote.c
+++ b/src/fu-remote.c
@@ -116,6 +116,10 @@ fu_remote_load_from_filename(FwupdRemote *self,
 		g_autofree gchar *tmp = g_key_file_get_string(kf, group, "Password", NULL);
 		fwupd_remote_set_password(self, tmp);
 	}
+	if (g_key_file_has_key(kf, group, "FirmwareBaseURI", NULL)) {
+		g_autofree gchar *tmp = g_key_file_get_string(kf, group, "FirmwareBaseURI", NULL);
+		fwupd_remote_set_firmware_base_uri(self, tmp);
+	}
 	if (g_key_file_has_key(kf, group, "OrderBefore", NULL)) {
 		g_autofree gchar *tmp = g_key_file_get_string(kf, group, "OrderBefore", NULL);
 		fwupd_remote_set_order_before(self, tmp);
@@ -197,6 +201,11 @@ fu_remote_save_to_filename(FwupdRemote *self,
 		g_key_file_set_string(kf, group, "Username", fwupd_remote_get_username(self));
 	if (fwupd_remote_get_password(self) != NULL)
 		g_key_file_set_string(kf, group, "Password", fwupd_remote_get_password(self));
+	if (fwupd_remote_get_firmware_base_uri(self) != NULL)
+		g_key_file_set_string(kf,
+				      group,
+				      "FirmwareBaseURI",
+				      fwupd_remote_get_firmware_base_uri(self));
 	if (fwupd_remote_get_order_after(self) != NULL) {
 		g_autofree gchar *str = g_strjoinv(";", fwupd_remote_get_order_after(self));
 		g_key_file_set_string(kf, group, "OrderAfter", str);
diff --git a/src/fu-util-common.c b/src/fu-util-common.c
index 0ad3bd0a72b4..596ce5577a3d 100644
--- a/src/fu-util-common.c
+++ b/src/fu-util-common.c
@@ -2121,6 +2121,11 @@ fu_util_remote_to_string(FwupdRemote *remote, guint idt)
 				  /* TRANSLATORS: remote URI */
 				  _("Metadata Signature"),
 				  fwupd_remote_get_metadata_uri_sig(remote));
+	fwupd_codec_string_append(str,
+				  idt + 1,
+				  /* TRANSLATORS: remote URI */
+				  _("Firmware Base URI"),
+				  fwupd_remote_get_firmware_base_uri(remote));
 	tmp = fwupd_remote_get_report_uri(remote);
 	if (tmp != NULL) {
 		/* TRANSLATORS: URI to send success/failure reports */