Repository: conan-io/conan
Pull request: Metadata files upload/download
Changelog: Feature: New metadata files associated to recipes and packages that can be uploaded, downloaded and added after the package exists.
Docs: Omit

This proposal implements upload + download flows, but not ``conan install`` or ``conan create`` installation of metadata. That would add too much extra complexity to the code and uncertainty and fragile developer flows, like metadata not being pulled from server because the package is already in the cache, despite users trying to ``conan install --metadata=*``, thinking that it will pull new metadata from servers.

So probably the best flow to support full-graph download of metadata would be:
- ``conan install ... --format=json > graph.json``
- convert ``graph.json`` => ``pkg_list.json``
- ``conan download --list=pkg_list.json --metadata=logs*``

This flow has been already implemented in https://github.com/conan-io/conan/pull/13928


Close https://github.com/conan-io/conan/issues/13715
Close https://github.com/conan-io/conan/issues/4610




Problem statement:
Embed test_package in the recipe
Provide a way to avoid having a `test_package` folder. Maybe declaring the contents of a `main.c/cpp` in the recipe could be enough.  It would be nice because:

- You could contribute with only a conanfile.py to a library, without polluting the repository.
- The `conan install --build` could run the test_package too.

To brainstorm:

- New conanfile method `test_package`? calling some helpers that will generate on the flight the test package process? Or should it be only a standard variable name containing the c/c++ code?
- Should this mechanism will be compatible or support having the classic `test_package` folder?
- Should accept a path to a file in case the repository already have an example?

[question] Test reports
### What is your question?

Dear Conan Folks,

I am a bit divided about the topic where shall test reports for a package belong to.
In this case for every package I have number of tests: Unit tests, static code analysis, functional tests, simulation tests, on device tests. Each of them will generate a report. I think that this shall be shipped along with the binaries or libraries of the package so that the binary and the reports are all in the same unit and can be referenced by the consumer of the package. But what I am concerened about is :
- The size of the package will increase.
Other alternatives that I thought about is:
- Having a generic repo for reports but then that will come with a disadvantage of linking the reports belonging to a package.
-  Having another conan package for storing reports again there also we need to link the reports  belonging to a package.
Could you please help me what is the suggestion from the experts?

Thanks!
Br,
Maitrey

### Have you read the CONTRIBUTING guide?

- [X] I've read the CONTRIBUTING guide

Hints from issue discussion:
Sometimes we use an example or test from the author. We just copy the original file to test_package.

We could provide a new helper to point that specific file, build (generate cmake, link conan libs) and run.
More things to discuss:

- Does it build the test in the cache ``test/hash`` folder? I'd say yes.
- CMake as only build system for this feature?
Only to increment some ideas, how Conda and VPKG test their packages?

- Conda uses a field in the recipe with the steps. But it's really shallow. For example, for zlib, they only check if the library has been created:
https://github.com/conda-forge/zlib-feedstock/blob/master/recipe/meta.yaml#L27

- VPKG uses the build as test. So if your recipes is able to be built on CI, probably it's okay.


We don't plan to embed test_package in the recipe, but the "metadata" feature allows to do this, the tests in https://github.com/conan-io/conan/pull/13918 covers this use case.
Hi @maitrey 

Thanks very much for your question.
I think it is very interesting, and I think we are already contemplating it, please check https://github.com/conan-io/conan/pull/13330

The idea is a metadata layer, that allows storing such reports, in the server, together with the package, but they will not be downloaded everytime the package is used. The storage in the server will still happen, that is inevitable, no matter the technology or the approach, it is necessary to store it, but the metadata layer will handle it automatically and make it very convenient.

We will continue finalizing this feature in the next releases and documenting it (you might experiment with it, but note it is not ready for production. Also, do not use if you are still using Conan 1.X; as until Conan 1.60, it might break the Conan 1.X client)
Thanks @memsharded for the excellent news. I am so glad that this feature is already being considered. Should I leave the question open meanwhile?
Yes, it can be open until we do the public release. Thanks!
The metadata story is implemented in PR https://github.com/conan-io/conan/pull/13918 for next 2.0.7

Commit message: wip

Patch to review:
diff --git a/conan/api/subapi/cache.py b/conan/api/subapi/cache.py
index 11f0b0073b4..d1d637fc3fe 100644
--- a/conan/api/subapi/cache.py
+++ b/conan/api/subapi/cache.py
@@ -19,6 +19,12 @@ def export_path(self, ref: RecipeReference):
         ref_layout = app.cache.ref_layout(ref)
         return ref_layout.export()
 
+    def recipe_metadata_path(self, ref: RecipeReference):
+        app = ConanApp(self.conan_api.cache_folder)
+        ref = _resolve_latest_ref(app, ref)
+        ref_layout = app.cache.ref_layout(ref)
+        return ref_layout.metadata()
+
     def export_source_path(self, ref: RecipeReference):
         app = ConanApp(self.conan_api.cache_folder)
         ref = _resolve_latest_ref(app, ref)
@@ -37,6 +43,12 @@ def build_path(self, pref: PkgReference):
         ref_layout = app.cache.pkg_layout(pref)
         return ref_layout.build()
 
+    def package_metadata_path(self, pref: PkgReference):
+        app = ConanApp(self.conan_api.cache_folder)
+        pref = _resolve_latest_pref(app, pref)
+        ref_layout = app.cache.pkg_layout(pref)
+        return ref_layout.metadata()
+
     def package_path(self, pref: PkgReference):
         app = ConanApp(self.conan_api.cache_folder)
         pref = _resolve_latest_pref(app, pref)
diff --git a/conan/api/subapi/download.py b/conan/api/subapi/download.py
index f2353b8b7b0..b364cb2aed6 100644
--- a/conan/api/subapi/download.py
+++ b/conan/api/subapi/download.py
@@ -12,16 +12,18 @@ class DownloadAPI:
     def __init__(self, conan_api):
         self.conan_api = conan_api
 
-    def recipe(self, ref: RecipeReference, remote: Remote):
+    def recipe(self, ref: RecipeReference, remote: Remote, metadata=None):
         output = ConanOutput()
         app = ConanApp(self.conan_api.cache_folder)
         skip_download = app.cache.exists_rrev(ref)
         if skip_download:
             output.info(f"Skip recipe {ref.repr_notime()} download, already in cache")
+            if metadata:
+                app.remote_manager.get_recipe_metadata(ref, remote, metadata)
             return False
 
         output.info(f"Downloading recipe '{ref.repr_notime()}'")
-        app.remote_manager.get_recipe(ref, remote)
+        app.remote_manager.get_recipe(ref, remote, metadata)
 
         layout = app.cache.ref_layout(ref)
         conan_file_path = layout.conanfile()
@@ -32,7 +34,7 @@ def recipe(self, ref: RecipeReference, remote: Remote):
         retrieve_exports_sources(app.remote_manager, layout, conanfile, ref, [remote])
         return True
 
-    def package(self, pref: PkgReference, remote: Remote):
+    def package(self, pref: PkgReference, remote: Remote, metadata=None):
         output = ConanOutput()
         app = ConanApp(self.conan_api.cache_folder)
         if not app.cache.exists_rrev(pref.ref):
@@ -42,6 +44,8 @@ def package(self, pref: PkgReference, remote: Remote):
         skip_download = app.cache.exists_prev(pref)
         if skip_download:
             output.info(f"Skip package {pref.repr_notime()} download, already in cache")
+            if metadata:
+                app.remote_manager.get_package_metadata(pref, remote, metadata)
             return False
         layout = app.cache.ref_layout(pref.ref)
         conan_file_path = layout.conanfile()
diff --git a/conan/api/subapi/upload.py b/conan/api/subapi/upload.py
index 6d44cb92e22..9fb3ebe8318 100644
--- a/conan/api/subapi/upload.py
+++ b/conan/api/subapi/upload.py
@@ -2,6 +2,7 @@
 
 from conan.api.output import ConanOutput
 from conan.internal.conan_app import ConanApp
+from conan.internal.upload_metadata import gather_metadata
 from conans.client.cmd.uploader import PackagePreparator, UploadExecutor, UploadUpstreamChecker
 from conans.client.downloaders.download_cache import DownloadCache
 from conans.client.pkg_sign import PkgSignaturesPlugin
@@ -29,13 +30,19 @@ def check_upstream(self, package_list, remote, force=False):
 
         UploadUpstreamChecker(app).check(package_list, remote, force)
 
-    def prepare(self, package_list, enabled_remotes):
+    def prepare(self, package_list, enabled_remotes, metadata=None):
         """Compress the recipes and packages and fill the upload_data objects
         with the complete information. It doesn't perform the upload nor checks upstream to see
-        if the recipe is still there"""
+        if the recipe is still there
+        :param package_list:
+        :param enabled_remotes:
+        :param metadata: A list of patterns of metadata that should be uploaded. Default None
+                         means all metadata will be uploaded together with the pkg artifacts
+        """
         app = ConanApp(self.conan_api.cache_folder)
         preparator = PackagePreparator(app)
         preparator.prepare(package_list, enabled_remotes)
+        gather_metadata(package_list, app.cache, metadata)
         signer = PkgSignaturesPlugin(app.cache)
         # This might add files entries to package_list with signatures
         signer.sign(package_list)
diff --git a/conan/cli/commands/cache.py b/conan/cli/commands/cache.py
index aa2cbf973e8..582636cf797 100644
--- a/conan/cli/commands/cache.py
+++ b/conan/cli/commands/cache.py
@@ -21,7 +21,7 @@ def cache_path(conan_api: ConanAPI, parser, subparser, *args):
     Show the path to the Conan cache for a given reference.
     """
     subparser.add_argument("reference", help="Recipe reference or Package reference")
-    subparser.add_argument("--folder", choices=['export_source', 'source', 'build'],
+    subparser.add_argument("--folder", choices=['export_source', 'source', 'build', 'metadata'],
                            help="Path to show. The 'build'"
                                 " requires a package reference. If not specified it shows 'exports'"
                                 " path ")
@@ -40,6 +40,8 @@ def cache_path(conan_api: ConanAPI, parser, subparser, *args):
             path = conan_api.cache.export_source_path(ref)
         elif args.folder == "source":
             path = conan_api.cache.source_path(ref)
+        elif args.folder == "metadata":
+            path = conan_api.cache.recipe_metadata_path(ref)
         else:
             raise ConanException(f"'--folder {args.folder}' requires a valid package reference")
     else:
@@ -47,6 +49,8 @@ def cache_path(conan_api: ConanAPI, parser, subparser, *args):
             path = conan_api.cache.package_path(pref)
         elif args.folder == "build":
             path = conan_api.cache.build_path(pref)
+        elif args.folder == "metadata":
+            path = conan_api.cache.package_metadata_path(pref)
         else:
             raise ConanException(f"'--folder {args.folder}' requires a recipe reference")
     return path
@@ -64,7 +68,7 @@ def cache_clean(conan_api: ConanAPI, parser, subparser, *args):
     subparser.add_argument("-b", "--build", action='store_true', default=False,
                            help="Clean build folders")
     subparser.add_argument("-d", "--download", action='store_true', default=False,
-                           help="Clean download folders")
+                           help="Clean download and metadata folders")
     subparser.add_argument("-t", "--temp", action='store_true', default=False,
                            help="Clean temporary folders")
     subparser.add_argument('-p', '--package-query', action=OnceArgument,
diff --git a/conan/cli/commands/download.py b/conan/cli/commands/download.py
index 8fa7cd1e80f..1e819df5513 100644
--- a/conan/cli/commands/download.py
+++ b/conan/cli/commands/download.py
@@ -32,6 +32,9 @@ def download(conan_api: ConanAPI, parser, *args):
                              "(arch=x86 OR compiler=gcc)")
     parser.add_argument("-r", "--remote", action=OnceArgument, required=True,
                         help='Download from this specific remote')
+    parser.add_argument("-m", "--metadata", action='append',
+                        help='Download the metadata matching the pattern, even if the package is '
+                             'already in the cache and not downloaded')
     parser.add_argument("-l", "--list", help="Package list file")
 
     args = parser.parse_args(*args)
@@ -64,9 +67,9 @@ def download(conan_api: ConanAPI, parser, *args):
 
     if parallel <= 1:
         for ref in refs:
-            conan_api.download.recipe(ref, remote)
+            conan_api.download.recipe(ref, remote, args.metadata)
         for pref in prefs:
-            conan_api.download.package(pref, remote)
+            conan_api.download.package(pref, remote, args.metadata)
     else:
         _download_parallel(parallel, conan_api, refs, prefs, remote)
 
diff --git a/conan/cli/commands/upload.py b/conan/cli/commands/upload.py
index e395480ad41..cdb30963b7b 100644
--- a/conan/cli/commands/upload.py
+++ b/conan/cli/commands/upload.py
@@ -58,6 +58,9 @@ def upload(conan_api: ConanAPI, parser, *args):
     parser.add_argument('--dry-run', default=False, action='store_true',
                         help='Do not execute the real upload (experimental)')
     parser.add_argument("-l", "--list", help="Package list file")
+    parser.add_argument("-m", "--metadata", action='append',
+                        help='Upload the metadata, even if the package is already in the server and '
+                             'not uploaded')
 
     args = parser.parse_args(*args)
 
@@ -88,7 +91,7 @@ def upload(conan_api: ConanAPI, parser, *args):
     if not args.list and not args.confirm and "*" in args.pattern:
         _ask_confirm_upload(conan_api, package_list)
 
-    conan_api.upload.prepare(package_list, enabled_remotes)
+    conan_api.upload.prepare(package_list, enabled_remotes, args.metadata)
 
     if not args.dry_run:
         conan_api.upload.upload(package_list, remote)
diff --git a/conan/internal/upload_metadata.py b/conan/internal/upload_metadata.py
new file mode 100644
index 00000000000..356a3e81f6b
--- /dev/null
+++ b/conan/internal/upload_metadata.py
@@ -0,0 +1,38 @@
+import fnmatch
+import os
+
+from conan.api.output import ConanOutput
+
+
+def _metadata_files(folder, metadata):
+    result = {}
+    for root, _, files in os.walk(folder):
+        for f in files:
+            abs_path = os.path.join(root, f)
+            relpath = os.path.relpath(abs_path, folder)
+            if metadata:
+                if not any(fnmatch.fnmatch(relpath, m) for m in metadata):
+                    continue
+            path = os.path.join("metadata", relpath).replace("\\", "/")
+            result[path] = abs_path
+    return result
+
+
+def gather_metadata(upload_data, cache, metadata):
+    for rref, recipe_bundle in upload_data.refs():
+        if metadata or recipe_bundle["upload"]:
+            metadata_folder = cache.ref_layout(rref).metadata()
+            files = _metadata_files(metadata_folder, metadata)
+            if files:
+                ConanOutput(scope=str(rref)).info(f"Recipe metadata: {len(files)} files")
+                recipe_bundle.setdefault("files", {}).update(files)
+                recipe_bundle["upload"] = True
+
+        for pref, pkg_bundle in upload_data.prefs(rref, recipe_bundle):
+            if metadata or pkg_bundle["upload"]:
+                metadata_folder = cache.pkg_layout(pref).metadata()
+                files = _metadata_files(metadata_folder, metadata)
+                if files:
+                    ConanOutput(scope=str(pref)).info(f"Package metadata: {len(files)} files")
+                    pkg_bundle.setdefault("files", {}).update(files)
+                    pkg_bundle["upload"] = True
diff --git a/conans/client/remote_manager.py b/conans/client/remote_manager.py
index e39015dd281..55a430c71c4 100644
--- a/conans/client/remote_manager.py
+++ b/conans/client/remote_manager.py
@@ -39,7 +39,7 @@ def upload_package(self, pref, files_to_upload, remote):
         assert pref.revision, "upload_package requires PREV"
         self._call_remote(remote, "upload_package", pref, files_to_upload)
 
-    def get_recipe(self, ref, remote):
+    def get_recipe(self, ref, remote, metadata=None):
         """
         Read the conans from remotes
         Will iterate the remotes to find the conans unless remote was specified
@@ -53,7 +53,9 @@ def get_recipe(self, ref, remote):
 
         download_export = layout.download_export()
         try:
-            zipped_files = self._call_remote(remote, "get_recipe", ref, download_export)
+            zipped_files = self._call_remote(remote, "get_recipe", ref, download_export, metadata,
+                                             only_metadata=False)
+            # TODO: Optimize this call, it is slow to always query all revisions
             remote_refs = self._call_remote(remote, "get_recipe_revisions_references", ref)
             ref_time = remote_refs[0].timestamp
             ref.timestamp = ref_time
@@ -83,6 +85,23 @@ def get_recipe(self, ref, remote):
         # Make sure that the source dir is deleted
         rmdir(layout.source())
 
+    def get_recipe_metadata(self, ref, remote, metadata):
+        """
+        Get only the metadata for a locally existing recipe in Cache
+        """
+        assert ref.revision, "get_recipe without revision specified"
+        output = ConanOutput(scope=str(ref))
+        output.info("Retrieving recipe metadata from remote '%s' " % remote.name)
+        layout = self._cache.ref_layout(ref)
+        download_export = layout.download_export()
+        try:
+            self._call_remote(remote, "get_recipe", ref, download_export, metadata,
+                              only_metadata=True)
+        except BaseException:  # So KeyboardInterrupt also cleans things
+
+            output.error(f"Error downloading metadata from remote '{remote.name}'")
+            raise
+
     def get_recipe_sources(self, ref, layout, remote):
         assert ref.revision, "get_recipe_sources requires RREV"
 
@@ -96,7 +115,7 @@ def get_recipe_sources(self, ref, layout, remote):
         tgz_file = zipped_files[EXPORT_SOURCES_TGZ_NAME]
         uncompress_file(tgz_file, export_sources_folder, scope=str(ref))
 
-    def get_package(self, conanfile, pref, remote):
+    def get_package(self, conanfile, pref, remote, metadata=None):
         conanfile.output.info("Retrieving package %s from remote '%s' " % (pref.package_id,
                                                                            remote.name))
 
@@ -105,15 +124,35 @@ def get_package(self, conanfile, pref, remote):
         pkg_layout = self._cache.get_or_create_pkg_layout(pref)
         pkg_layout.package_remove()  # Remove first the destination folder
         with pkg_layout.set_dirty_context_manager():
-            self._get_package(pkg_layout, pref, remote, conanfile.output)
+            self._get_package(pkg_layout, pref, remote, conanfile.output, metadata)
+
+    def get_package_metadata(self, pref, remote, metadata):
+        """
+        only download the metadata, not the packge itself
+        """
+        output = ConanOutput(scope=str(pref.ref))
+        output.info("Retrieving package metadata %s from remote '%s' "
+                    % (pref.package_id, remote.name))
+
+        assert pref.revision is not None
+        pkg_layout = self._cache.pkg_layout(pref)
+        try:
+            download_pkg_folder = pkg_layout.download_package()
+            self._call_remote(remote, "get_package", pref, download_pkg_folder,
+                              metadata, only_metadata=True)
+        except BaseException as e:  # So KeyboardInterrupt also cleans things
+            output.error("Exception while getting package metadata: %s" % str(pref.package_id))
+            output.error("Exception: %s %s" % (type(e), str(e)))
+            raise
 
-    def _get_package(self, layout, pref, remote, scoped_output):
+    def _get_package(self, layout, pref, remote, scoped_output, metadata):
         try:
             assert pref.revision is not None
 
             download_pkg_folder = layout.download_package()
             # Download files to the pkg_tgz folder, not to the final one
-            zipped_files = self._call_remote(remote, "get_package", pref, download_pkg_folder)
+            zipped_files = self._call_remote(remote, "get_package", pref, download_pkg_folder,
+                                             metadata, only_metadata=False)
             zipped_files = {k: v for k, v in zipped_files.items() if not k.startswith(METADATA)}
             # quick server package integrity check:
             for f in ("conaninfo.txt", "conanmanifest.txt", "conan_package.tgz"):
diff --git a/conans/client/rest/rest_client.py b/conans/client/rest/rest_client.py
index ebfe2505503..ab26bde4079 100644
--- a/conans/client/rest/rest_client.py
+++ b/conans/client/rest/rest_client.py
@@ -60,14 +60,14 @@ def _get_api(self):
                              self._requester, self._config, self._verify_ssl,
                              checksum_deploy)
 
-    def get_recipe(self, ref, dest_folder):
-        return self._get_api().get_recipe(ref, dest_folder)
+    def get_recipe(self, ref, dest_folder, metadata, only_metadata):
+        return self._get_api().get_recipe(ref, dest_folder, metadata, only_metadata)
 
     def get_recipe_sources(self, ref, dest_folder):
         return self._get_api().get_recipe_sources(ref, dest_folder)
 
-    def get_package(self, pref, dest_folder):
-        return self._get_api().get_package(pref, dest_folder)
+    def get_package(self, pref, dest_folder, metadata, only_metadata):
+        return self._get_api().get_package(pref, dest_folder, metadata, only_metadata)
 
     def upload_recipe(self, ref, files_to_upload):
         return self._get_api().upload_recipe(ref, files_to_upload)
diff --git a/conans/client/rest/rest_client_v2.py b/conans/client/rest/rest_client_v2.py
index ce030be090b..542a0d2299f 100644
--- a/conans/client/rest/rest_client_v2.py
+++ b/conans/client/rest/rest_client_v2.py
@@ -1,4 +1,5 @@
 import copy
+import fnmatch
 import os
 
 from conan.api.output import ConanOutput
@@ -35,12 +36,17 @@ def _get_file_list_json(self, url):
         data["files"] = list(d.replace("\\", "/") for d in data["files"].keys())
         return data
 
-    def get_recipe(self, ref, dest_folder):
+    def get_recipe(self, ref, dest_folder, metadata, only_metadata):
         url = self.router.recipe_snapshot(ref)
         data = self._get_file_list_json(url)
         files = data["files"]
         accepted_files = ["conanfile.py", "conan_export.tgz", "conanmanifest.txt", "metadata/sign"]
-        files = [f for f in files if any(f.startswith(m) for m in accepted_files)]
+        if only_metadata:
+            accepted_files = []
+        metadata = metadata or []
+        metadata = [f"metadata/{m}" for m in metadata]
+        files = [f for f in files if any(f.startswith(m) for m in accepted_files)
+                 or any(fnmatch.fnmatch(f, m) for m in metadata)]
 
         # If we didn't indicated reference, server got the latest, use absolute now, it's safer
         urls = {fn: self.router.recipe_file(ref, fn) for fn in files}
@@ -65,13 +71,19 @@ def get_recipe_sources(self, ref, dest_folder):
         ret = {fn: os.path.join(dest_folder, fn) for fn in files}
         return ret
 
-    def get_package(self, pref, dest_folder):
+    def get_package(self, pref, dest_folder, metadata, only_metadata):
         url = self.router.package_snapshot(pref)
         data = self._get_file_list_json(url)
         files = data["files"]
         # Download only known files, but not metadata (except sign)
         accepted_files = ["conaninfo.txt", "conan_package.tgz", "conanmanifest.txt", "metadata/sign"]
-        files = [f for f in files if any(f.startswith(m) for m in accepted_files)]
+        if only_metadata:
+            accepted_files = []
+        metadata = metadata or []
+        metadata = [f"metadata/{m}" for m in metadata]
+        files = [f for f in files if any(f.startswith(m) for m in accepted_files)
+                 or any(fnmatch.fnmatch(f, m) for m in metadata)]
+
         # If we didn't indicated reference, server got the latest, use absolute now, it's safer
         urls = {fn: self.router.package_file(pref, fn) for fn in files}
         self._download_and_save_files(urls, dest_folder, files, scope=str(pref.ref))
diff --git a/conans/test/integration/metadata/test_metadata_commands.py b/conans/test/integration/metadata/test_metadata_commands.py
new file mode 100644
index 00000000000..c4eec4ffda1
--- /dev/null
+++ b/conans/test/integration/metadata/test_metadata_commands.py
@@ -0,0 +1,59 @@
+import os
+
+from conans.test.assets.genconanfile import GenConanfile
+from conans.test.utils.tools import TestClient
+from conans.util.files import save
+
+
+class TestMetadataCommands:
+
+    def test_upload(self):
+        c = TestClient(default_server_user=True)
+        c.save({"conanfile.py": GenConanfile("pkg", "0.1"),
+                "myfile": "mycontent",
+                "folder/other": "other content"})
+        c.run("create .")
+        pid = c.created_package_id("pkg/0.1")
+
+        # Add some metadata
+        c.run("cache path pkg/0.1 --folder=metadata")
+        metadata_path = str(c.stdout).strip()
+        myfile = os.path.join(metadata_path, "logs", "mylogs.txt")
+        save(myfile, "mylogs!!!!")
+
+        c.run(f"cache path pkg/0.1:{pid} --folder=metadata")
+        pkg_metadata_path = str(c.stdout).strip()
+        myfile = os.path.join(pkg_metadata_path, "logs", "mybuildlogs.txt")
+        save(myfile, "mybuildlogs!!!!")
+
+        # Now upload everything
+        c.run("upload * -c -r=default")
+        assert "pkg/0.1: Recipe metadata: 1 files" in c.out
+        assert "pkg/0.1:da39a3ee5e6b4b0d3255bfef95601890afd80709: Package metadata: 1 files" in c.out
+
+        # Add new files to the metadata
+        myfile = os.path.join(metadata_path, "logs", "mylogs2.txt")
+        save(myfile, "mylogs2!!!!")
+        myfile = os.path.join(pkg_metadata_path, "logs", "mybuildlogs2.txt")
+        save(myfile, "mybuildlogs2!!!!")
+        # Upload the metadata, even if the revisions exist in the server
+        # adding the new metadata logs files
+        c.run("upload * -c -r=default --metadata=*")
+        assert "pkg/0.1: Recipe metadata: 2 files" in c.out
+        assert "pkg/0.1:da39a3ee5e6b4b0d3255bfef95601890afd80709: Package metadata: 2 files" in c.out
+
+        c.run("remove * -c")
+        c.run("install --requires=pkg/0.1")  # wont install metadata by default
+        c.run("cache path pkg/0.1 --folder=metadata")
+        metadata_path = str(c.stdout).strip()
+        c.run(f"cache path pkg/0.1:{pid} --folder=metadata")
+        pkg_metadata_path = str(c.stdout).strip()
+        assert not os.path.exists(metadata_path)
+        assert not os.path.exists(pkg_metadata_path)
+
+        # Forcing the download of the metadata of cache-existing things with the "download" command
+        c.run("download pkg/0.1 -r=default --metadata=*")
+        for f in "logs/mylogs.txt", "logs/mylogs2.txt":
+            assert os.path.isfile(os.path.join(metadata_path, f))
+        for f in "logs/mybuildlogs.txt", "logs/mybuildlogs2.txt":
+            assert os.path.isfile(os.path.join(pkg_metadata_path, f))
diff --git a/conans/test/integration/metadata/test_metadata_test_package.py b/conans/test/integration/metadata/test_metadata_test_package.py
new file mode 100644
index 00000000000..25f051ca489
--- /dev/null
+++ b/conans/test/integration/metadata/test_metadata_test_package.py
@@ -0,0 +1,51 @@
+import os
+import shutil
+import textwrap
+
+from conans.test.assets.genconanfile import GenConanfile
+from conans.test.utils.tools import TestClient
+from conans.util.files import save
+
+
+class TestMetadataTestPackage:
+    """ It is possible to store the test_package itself in the recipe metadata and recover it
+    later to execute it
+    """
+
+    def test_round_trip_with_hook(self):
+        c = TestClient(default_server_user=True)
+        # TODO: Better strategy for storing clean test_package, zipping it, etc
+        my_hook = textwrap.dedent("""\
+            import os
+            from conan.tools.files import copy
+
+            def post_export(conanfile):
+               conanfile.output.info("Storing test_package")
+               folder = os.path.join(conanfile.recipe_folder, "test_package")
+               copy(conanfile, "*", src=folder,
+                    dst=os.path.join(conanfile.recipe_metadata_folder, "test_package"))
+            """)
+        hook_path = os.path.join(c.cache.hooks_path, "my_hook", "hook_my_hook.py")
+        save(hook_path, my_hook)
+        c.save({"conanfile.py": GenConanfile("pkg", "0.1"),
+                "test_package/conanfile.py": GenConanfile().with_test("pass")})
+        c.run("create .")
+        assert "Testing the package" in c.out
+
+        # Now upload and remove everything
+        c.run("upload * -c -r=default")
+        assert "pkg/0.1: Recipe metadata: 1 files" in c.out
+        c.run("remove * -c")
+
+        c.run("install --requires=pkg/0.1 -r=default")
+        # Recovery of the test package
+        # TODO: Discuss if we want better UX, in a single step or something like that
+        # Forcing the download of the metadata of cache-existing things with the "download" command
+        c.run("download pkg/0.1 -r=default --metadata=test_package*")
+        c.run("cache path pkg/0.1 --folder=metadata")
+        metadata_path = str(c.stdout).strip()
+        shutil.copytree(metadata_path, os.path.join(c.current_folder, "metadata"))
+
+        # Execute the test_package
+        c.run("test metadata/test_package pkg/0.1")
+        assert "pkg/0.1 (test package): Running test()" in c.out
diff --git a/conans/test/integration/remote/rest_api_test.py b/conans/test/integration/remote/rest_api_test.py
index 1fbb30567c6..eab9fb8b3d0 100644
--- a/conans/test/integration/remote/rest_api_test.py
+++ b/conans/test/integration/remote/rest_api_test.py
@@ -80,7 +80,7 @@ def test_get_conan(self):
 
         # Get the conans
         tmp_dir = temp_folder()
-        self.api.get_recipe(ref, tmp_dir)
+        self.api.get_recipe(ref, tmp_dir, metadata=None, only_metadata=False)
         self.assertIn(CONANFILE, os.listdir(tmp_dir))
         self.assertIn(CONAN_MANIFEST, os.listdir(tmp_dir))
 
@@ -95,7 +95,7 @@ def test_get_package(self):
 
         # Get the package
         tmp_dir = temp_folder()
-        self.api.get_package(pref, tmp_dir)
+        self.api.get_package(pref, tmp_dir, metadata=None, only_metadata=False)
         # The hello.cpp file is not downloaded!
         self.assertNotIn("hello.cpp", os.listdir(tmp_dir))
 
@@ -104,7 +104,7 @@ def test_upload_huge_conan(self):
         self._upload_recipe(ref, {"file9.cpp": ""})
 
         tmp = temp_folder()
-        files = self.api.get_recipe(ref, tmp)
+        files = self.api.get_recipe(ref, tmp, metadata=None, only_metadata=False)
         self.assertIsNotNone(files)
         self.assertFalse(os.path.exists(os.path.join(tmp, "file9.cpp")))
 
diff --git a/conans/test/integration/remote/token_refresh_test.py b/conans/test/integration/remote/token_refresh_test.py
index afa231c7120..8d3fef278ad 100644
--- a/conans/test/integration/remote/token_refresh_test.py
+++ b/conans/test/integration/remote/token_refresh_test.py
@@ -92,7 +92,8 @@ def test_auth_with_token(self):
         with mock.patch("conans.client.rest.auth_manager.UserInput.request_login",
                         return_value=("myuser", "mypassword")):
 
-            self.auth_manager.call_rest_api_method(self.remote, "get_recipe", self.ref, ".")
+            self.auth_manager.call_rest_api_method(self.remote, "get_recipe", self.ref, ".",
+                                                   metadata=None, only_metadata=False)
             self.assertEqual(self.localdb.user, "myuser")
             self.assertEqual(self.localdb.access_token, "access_token")
             self.assertEqual(self.localdb.refresh_token, "refresh_token")
@@ -106,7 +107,8 @@ def test_refresh_with_token(self):
             self.localdb.access_token = "expired"
             self.localdb.refresh_token = "refresh_token"
 
-            self.auth_manager.call_rest_api_method(self.remote, "get_recipe", self.ref, ".")
+            self.auth_manager.call_rest_api_method(self.remote, "get_recipe", self.ref, ".",
+                                                   metadata=None, only_metadata=False)
             self.assertEqual(self.localdb.user, "myuser")
             self.assertEqual(self.localdb.access_token, "refreshed_access_token")
             self.assertEqual(self.localdb.refresh_token, "refresh_token")


Code review comments:
diff_hunk: @@ -0,0 +1,38 @@
+import fnmatch
+import os
+
+from conan.api.output import ConanOutput
+
+
+def _metadata_files(folder, metadata):
+    result = {}
+    for root, _, files in os.walk(folder):
+        for f in files:
+            abs_path = os.path.join(root, f)
+            relpath = os.path.relpath(abs_path, folder)
+            if metadata:
+                if not any(fnmatch.fnmatch(relpath, m) for m in metadata):
+                    continue
+            path = os.path.join("metadata", relpath).replace("\\", "/")
+            result[path] = abs_path
+    return result
+
+
+def gather_metadata(upload_data, cache, metadata):
path: conan/internal/upload_metadata.py
text: @user1:
I think it's clearer to replace `upload_data` by `pkg_list`

@author:
Done