Repository: microsoft/vcpkg-tool
Pull request #1607: [spdx] Replace CMake variables in `vcpkg_download_distfile()`
Author: login: Thomas1664
Created: 2025-03-08T09:58:55Z | Merged: 2025-03-31T22:17:32Z

Partially fixes https://github.com/microsoft/vcpkg/issues/44188
Fixes https://github.com/microsoft/vcpkg/issues/44040


Diff:
diff --git a/src/vcpkg-test/spdx.cpp b/src/vcpkg-test/spdx.cpp
index f3233f3381..f9dd28daf9 100644
--- a/src/vcpkg-test/spdx.cpp
+++ b/src/vcpkg-test/spdx.cpp
@@ -98,6 +98,264 @@ TEST_CASE ("extract arg from cmake invocation args", "[spdx]")
     }
 }
 
+TEST_CASE ("spdx run resource heuristics", "[spdx]")
+{
+    auto portfile_cmake = R"(
+vcpkg_download_distfile(ARCHIVE
+    URLS "https://vcpkg-download-distfile.dev/${VERSION}.tar.gz"
+         "https://vcpkg-download-distfile.dev/${VERSION}-other.tar.gz"
+    FILENAME "distfile-${VERSION}.tar.gz"
+    SHA512 distfile_test_1
+)
+vcpkg_from_github(
+    OUT_SOURCE_PATH SOURCE_PATH
+    REPO from/github
+    REF v${VERSION}
+    SHA512 from_github_test_1
+    HEAD_REF devel
+)
+vcpkg_from_gitlab(
+    OUT_SOURCE_PATH SOURCE_PATH
+    GITLAB_URL https://from.gitlab.org
+    REPO from/gitlab
+    REF "${VERSION}"
+    SHA512 from_gitlab_test_1
+)
+vcpkg_from_sourceforge(
+    OUT_SOURCE_PATH SOURCE_PATH
+    REPO sourceforge
+    REF sourceforge
+    FILENAME "sourceforge-${VERSION}.tar.gz"
+    SHA512 sourceforge_test_1
+    )
+vcpkg_from_bitbucket(
+    OUT_SOURCE_PATH SOURCE_PATH
+    REPO from/bitbucket
+    REF "v${VERSION}"
+    SHA512 from_bitbucket_test_1
+    HEAD_REF master
+)
+vcpkg_download_distfile(ARCHIVE
+    URLS "https://vcpkg-download-distfile.dev/${VERSION}.tar.gz"
+         "https://vcpkg-download-distfile.dev/${VERSION}-other.tar.gz"
+    FILENAME "distfile-${VERSION}.tar.gz"
+    SHA512 distfile_test_2
+)
+vcpkg_from_github(
+    OUT_SOURCE_PATH SOURCE_PATH
+    REPO from/github
+    REF v${VERSION}
+    SHA512 from_github_test_2
+    HEAD_REF devel
+)
+vcpkg_from_gitlab(
+    OUT_SOURCE_PATH SOURCE_PATH
+    GITLAB_URL https://from.gitlab.org
+    REPO from/gitlab
+    REF "${VERSION}"
+    SHA512 from_gitlab_test_2
+)
+vcpkg_from_sourceforge(
+    OUT_SOURCE_PATH SOURCE_PATH
+    REPO sourceforge
+    REF sourceforge
+    FILENAME "sourceforge-${VERSION}.tar.gz"
+    SHA512 sourceforge_test_2
+    )
+vcpkg_from_bitbucket(
+    OUT_SOURCE_PATH SOURCE_PATH
+    REPO from/bitbucket
+    REF "v${VERSION}"
+    SHA512 from_bitbucket_test_2
+    HEAD_REF master
+)
+vcpkg_from_git(
+    OUT_SOURCE_PATH SOURCE_PATH
+    URL https://from-git-1.dev
+    REF "${VERSION}"
+    HEAD_REF main
+)
+vcpkg_from_git(
+    OUT_SOURCE_PATH SOURCE_PATH
+    URL https://from-git-2.dev
+    REF "${VERSION}"
+    HEAD_REF main
+)
+    )";
+    auto expected = Json::parse(R"json(
+{
+  "packages": [
+    {
+      "SPDXID": "SPDXRef-resource-0",
+      "name": "from/github",
+      "downloadLocation": "git+https://github.com/from/github@v3.2.1",
+      "licenseConcluded": "NOASSERTION",
+      "licenseDeclared": "NOASSERTION",
+      "copyrightText": "NOASSERTION",
+      "checksums": [
+        {
+          "algorithm": "SHA512",
+          "checksumValue": "from_github_test_1"
+        }
+      ]
+    },
+    {
+      "SPDXID": "SPDXRef-resource-1",
+      "name": "from/github",
+      "downloadLocation": "git+https://github.com/from/github@v3.2.1",
+      "licenseConcluded": "NOASSERTION",
+      "licenseDeclared": "NOASSERTION",
+      "copyrightText": "NOASSERTION",
+      "checksums": [
+        {
+          "algorithm": "SHA512",
+          "checksumValue": "from_github_test_2"
+        }
+      ]
+    },
+    {
+      "SPDXID": "SPDXRef-resource-2",
+      "name": "from/gitlab",
+      "downloadLocation": "git+https://from.gitlab.org/from/gitlab@3.2.1",
+      "licenseConcluded": "NOASSERTION",
+      "licenseDeclared": "NOASSERTION",
+      "copyrightText": "NOASSERTION",
+      "checksums": [
+        {
+          "algorithm": "SHA512",
+          "checksumValue": "from_gitlab_test_1"
+        }
+      ]
+    },
+    {
+      "SPDXID": "SPDXRef-resource-3",
+      "name": "from/gitlab",
+      "downloadLocation": "git+https://from.gitlab.org/from/gitlab@3.2.1",
+      "licenseConcluded": "NOASSERTION",
+      "licenseDeclared": "NOASSERTION",
+      "copyrightText": "NOASSERTION",
+      "checksums": [
+        {
+          "algorithm": "SHA512",
+          "checksumValue": "from_gitlab_test_2"
+        }
+      ]
+    },
+    {
+      "SPDXID": "SPDXRef-resource-4",
+      "name": "https://from-git-1.dev",
+      "downloadLocation": "git+https://from-git-1.dev@3.2.1",
+      "licenseConcluded": "NOASSERTION",
+      "licenseDeclared": "NOASSERTION",
+      "copyrightText": "NOASSERTION"
+    },
+    {
+      "SPDXID": "SPDXRef-resource-5",
+      "name": "https://from-git-2.dev",
+      "downloadLocation": "git+https://from-git-2.dev@3.2.1",
+      "licenseConcluded": "NOASSERTION",
+      "licenseDeclared": "NOASSERTION",
+      "copyrightText": "NOASSERTION"
+    },
+    {
+      "SPDXID": "SPDXRef-resource-6",
+      "name": "distfile-3.2.1.tar.gz",
+      "packageFileName": "distfile-3.2.1.tar.gz",
+      "downloadLocation": "https://vcpkg-download-distfile.dev/3.2.1.tar.gz",
+      "licenseConcluded": "NOASSERTION",
+      "licenseDeclared": "NOASSERTION",
+      "copyrightText": "NOASSERTION",
+      "checksums": [
+        {
+          "algorithm": "SHA512",
+          "checksumValue": "distfile_test_1"
+        }
+      ]
+    },
+    {
+      "SPDXID": "SPDXRef-resource-7",
+      "name": "distfile-3.2.1.tar.gz",
+      "packageFileName": "distfile-3.2.1.tar.gz",
+      "downloadLocation": "https://vcpkg-download-distfile.dev/3.2.1.tar.gz",
+      "licenseConcluded": "NOASSERTION",
+      "licenseDeclared": "NOASSERTION",
+      "copyrightText": "NOASSERTION",
+      "checksums": [
+        {
+          "algorithm": "SHA512",
+          "checksumValue": "distfile_test_2"
+        }
+      ]
+    },
+    {
+      "SPDXID": "SPDXRef-resource-8",
+      "name": "sourceforge-3.2.1.tar.gz",
+      "packageFileName": "sourceforge-3.2.1.tar.gz",
+      "downloadLocation": "https://sourceforge.net/projects/sourceforge/files/sourceforge/sourceforge-3.2.1.tar.gz",
+      "licenseConcluded": "NOASSERTION",
+      "licenseDeclared": "NOASSERTION",
+      "copyrightText": "NOASSERTION",
+      "checksums": [
+        {
+          "algorithm": "SHA512",
+          "checksumValue": "sourceforge_test_1"
+        }
+      ]
+    },
+    {
+      "SPDXID": "SPDXRef-resource-9",
+      "name": "sourceforge-3.2.1.tar.gz",
+      "packageFileName": "sourceforge-3.2.1.tar.gz",
+      "downloadLocation": "https://sourceforge.net/projects/sourceforge/files/sourceforge/sourceforge-3.2.1.tar.gz",
+      "licenseConcluded": "NOASSERTION",
+      "licenseDeclared": "NOASSERTION",
+      "copyrightText": "NOASSERTION",
+      "checksums": [
+        {
+          "algorithm": "SHA512",
+          "checksumValue": "sourceforge_test_2"
+        }
+      ]
+    },
+    {
+      "SPDXID": "SPDXRef-resource-10",
+      "name": "from/bitbucket",
+      "downloadLocation": "git+https://bitbucket.com/from/bitbucket@v3.2.1",
+      "licenseConcluded": "NOASSERTION",
+      "licenseDeclared": "NOASSERTION",
+      "copyrightText": "NOASSERTION",
+      "checksums": [
+        {
+          "algorithm": "SHA512",
+          "checksumValue": "from_bitbucket_test_1"
+        }
+      ]
+    },
+    {
+      "SPDXID": "SPDXRef-resource-11",
+      "name": "from/bitbucket",
+      "downloadLocation": "git+https://bitbucket.com/from/bitbucket@v3.2.1",
+      "licenseConcluded": "NOASSERTION",
+      "licenseDeclared": "NOASSERTION",
+      "copyrightText": "NOASSERTION",
+      "checksums": [
+        {
+          "algorithm": "SHA512",
+          "checksumValue": "from_bitbucket_test_2"
+        }
+      ]
+    }
+  ]
+})json",
+                                "test")
+                        .value(VCPKG_LINE_INFO);
+
+    auto generated_spdx = run_resource_heuristics(portfile_cmake, "3.2.1");
+    auto spdx_str = Json::stringify(generated_spdx);
+    auto res = Json::parse(spdx_str, "test").value(VCPKG_LINE_INFO);
+    Test::check_json_eq(expected.value, res.value);
+}
+
 TEST_CASE ("spdx maximum serialization", "[spdx]")
 {
     PackagesDirAssigner packages_dir_assigner{"test_packages_root"};
diff --git a/src/vcpkg/spdx.cpp b/src/vcpkg/spdx.cpp
index ba77e2d144..6e7906866d 100644
--- a/src/vcpkg/spdx.cpp
+++ b/src/vcpkg/spdx.cpp
@@ -233,7 +233,7 @@ static void find_all_git(StringView text, Json::Array& packages, StringView vers
     }
 }
 
-static void find_all_distfile(StringView text, Json::Array& packages)
+static void find_all_distfile(StringView text, Json::Array& packages, StringView version_text)
 {
     auto it = text.begin();
     while (it != text.end())
@@ -244,11 +244,12 @@ static void find_all_distfile(StringView text, Json::Array& packages)
             it = text.end();
             continue;
         }
-        auto url = extract_arg_from_cmake_invocation_args(distfile, CMakeVariableUrls);
-        auto filename = extract_arg_from_cmake_invocation_args(distfile, CMakeVariableFilename);
+        auto url = fix_ref_version(extract_arg_from_cmake_invocation_args(distfile, CMakeVariableUrls), version_text);
+        auto filename =
+            fix_ref_version(extract_arg_from_cmake_invocation_args(distfile, CMakeVariableFilename), version_text);
         auto sha = extract_arg_from_cmake_invocation_args(distfile, CMakeVariableSHA512);
         packages.push_back(make_resource(
-            fmt::format("SPDXRef-resource-{}", packages.size()), filename, url.to_string(), sha, filename));
+            fmt::format("SPDXRef-resource-{}", packages.size()), filename, std::move(url), sha, filename));
         it = distfile.end();
     }
 }
@@ -266,7 +267,8 @@ static void find_all_sourceforge(StringView text, Json::Array& packages, StringV
         }
         auto repo = extract_arg_from_cmake_invocation_args(sfg, CMakeVariableRepo);
         auto ref = fix_ref_version(extract_arg_from_cmake_invocation_args(sfg, CMakeVariableRef), version_text);
-        auto filename = extract_arg_from_cmake_invocation_args(sfg, CMakeVariableFilename);
+        auto filename =
+            fix_ref_version(extract_arg_from_cmake_invocation_args(sfg, CMakeVariableFilename), version_text);
         auto sha = extract_arg_from_cmake_invocation_args(sfg, CMakeVariableSHA512);
         auto url = fmt::format("https://sourceforge.net/projects/{}/files/{}/{}", repo, ref, filename);
         packages.push_back(make_resource(
@@ -285,7 +287,7 @@ Json::Object vcpkg::run_resource_heuristics(StringView contents, StringView vers
     find_all_github(contents, packages, version_text);
     find_all_gitlab(contents, packages, version_text);
     find_all_git(contents, packages, version_text);
-    find_all_distfile(contents, packages);
+    find_all_distfile(contents, packages, version_text);
     find_all_sourceforge(contents, packages, version_text);
     find_all_bitbucket(contents, packages, version_text);
 


Reviews:
user: login: BillyONeal
body: Can you add a test? Would merge this with one.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-11T17:29:52Z
user: login: Thomas1664
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-22T09:55:44Z
user: login: BillyONeal
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-31T20:21:44Z

Review comments:
diff_hunk: @@ -244,11 +244,12 @@ static void find_all_distfile(StringView text, Json::Array& packages)
             it = text.end();
             continue;
         }
-        auto url = extract_arg_from_cmake_invocation_args(distfile, CMakeVariableUrls);
-        auto filename = extract_arg_from_cmake_invocation_args(distfile, CMakeVariableFilename);
+        auto url = fix_ref_version(extract_arg_from_cmake_invocation_args(distfile, CMakeVariableUrls), version_text);
+        auto filename =
+            fix_ref_version(extract_arg_from_cmake_invocation_args(distfile, CMakeVariableFilename), version_text);
         auto sha = extract_arg_from_cmake_invocation_args(distfile, CMakeVariableSHA512);
         packages.push_back(make_resource(
-            fmt::format("SPDXRef-resource-{}", packages.size()), filename, url.to_string(), sha, filename));
+            fmt::format("SPDXRef-resource-{}", packages.size()), filename, std::move(url), sha, filename));
path: src/vcpkg/spdx.cpp
user: login: BillyONeal
body: As far as I know fmt never moves from one of the args so this isn't a meaningful move.
created_at: 2025-03-11T17:28:57Z
author_association: MEMBER
diff_hunk: @@ -244,11 +244,12 @@ static void find_all_distfile(StringView text, Json::Array& packages)
             it = text.end();
             continue;
         }
-        auto url = extract_arg_from_cmake_invocation_args(distfile, CMakeVariableUrls);
-        auto filename = extract_arg_from_cmake_invocation_args(distfile, CMakeVariableFilename);
+        auto url = fix_ref_version(extract_arg_from_cmake_invocation_args(distfile, CMakeVariableUrls), version_text);
+        auto filename =
+            fix_ref_version(extract_arg_from_cmake_invocation_args(distfile, CMakeVariableFilename), version_text);
         auto sha = extract_arg_from_cmake_invocation_args(distfile, CMakeVariableSHA512);
         packages.push_back(make_resource(
-            fmt::format("SPDXRef-resource-{}", packages.size()), filename, url.to_string(), sha, filename));
+            fmt::format("SPDXRef-resource-{}", packages.size()), filename, std::move(url), sha, filename));
path: src/vcpkg/spdx.cpp
user: login: Thomas1664
body: The move is not inside format. url goes directly into `make_resource`
created_at: 2025-03-22T09:55:43Z
author_association: CONTRIBUTOR

PR comments:
user: login: Thomas1664
created_at: 2025-03-08T10:52:34Z
body: @BillyONeal I'm not sure how to handle the path issue:

From https://github.com/microsoft/vcpkg/issues/44188#issuecomment-2702968598:

> Also, Paths are also invalid:
>
>"fileName": ".//home/igor/development/vcpkg/vcpkg_original/ports/bzip2/bzip2.pc.in",

The code expects relative paths:

https://github.com/microsoft/vcpkg-tool/blob/d034830494d0a3779bc231f455a9d172fa319c66/src/vcpkg/spdx.cpp#L381-L386

but we feed in absolute paths by including the port directory:

https://github.com/microsoft/vcpkg-tool/blob/d034830494d0a3779bc231f455a9d172fa319c66/src/vcpkg/commands.build.cpp#L1247

https://github.com/microsoft/vcpkg-tool/blob/d034830494d0a3779bc231f455a9d172fa319c66/src/vcpkg/commands.build.cpp#L1331

https://github.com/microsoft/vcpkg-tool/blob/d034830494d0a3779bc231f455a9d172fa319c66/src/vcpkg/commands.build.cpp#L934-L938

However, I think we should be using absolute paths because we might lose the information of where a file comes from in case of overlay ports.

On the other hand, absolute paths are invalid after downloading from a binary cache on different machines.
author_association: CONTRIBUTOR
user: login: BillyONeal
created_at: 2025-03-10T18:01:12Z
body: >On the other hand, absolute paths are invalid after downloading from a binary cache on different machines.

Right, this is why the paths have to be relative.

>However, I think we should be using absolute paths because we might lose the information of where a file comes from in case of overlay ports.

That's why the SHAs of the files are also in the SPDX. It isn't there's a hard link between any particular absolute path and anything else, any more than there is between that relative path and anything else.
author_association: MEMBER
user: login: Thomas1664
created_at: 2025-03-22T11:41:00Z
body: > Can you add a test? Would merge this with one.

Hi @BillyONeal,

I added a unit test to check the correct replacement of `${VERSION}` in all CMake functions and to ensure that all instances of those functions are found.

As you proposed, I didn't fix the absolute path issue mentioned in https://github.com/microsoft/vcpkg/issues/44188. Therefore, plese don't close that issue yet - I will fix the absolute path issue in a different PR.
author_association: CONTRIBUTOR