Repository: conan-io/conan

Issue:
[question] Should sources backup feature work with "conan source"?
### What is your question?

Hi, I'm trying to start using sources backup feature. I've noticed that it doesn't upload sources that were downloaded through "conan source" call. Is this the intended behavior?

My workflow:
1) conan export on recipe folder
2) conan source with same version
3) conan upload

I was expecting that this will allow me to backup sources without need to call conan create. But conan says
```
-------- Uploading backup sources --------
No backup sources files to upload
```
I've looked at folder with sources and it has downloaded archive, but it doesn't have "references" set so probably that's why it doesn't upload them together with the recipe. Should this work or is this the expected behavior?

```
{"references": {"unknown": ["https://ftpmirror.gnu.org/gnu/libiconv/libiconv-1.18.tar.gz", "https://ftp.gnu.org/gnu/libiconv/libiconv-1.18.tar.gz"]}, "timestamp": 1752675688}
```

### Have you read the CONTRIBUTING guide?

- [x] I've read the CONTRIBUTING guide


Discussion:
Hi @Nekto89 

Thanks for your question.

The ``conan upload`` command uploads conan packages and its associated ``backup-sources`` if they have any.
As you are not uploading any packages, no sources will be uploaded.

You can use the ``conan cache backup-upload`` command instead to upload backup sources decoupled from uploading packages.
> You can use the `conan cache backup-upload` command instead to upload backup sources decoupled from uploading packages.

But then I would need to clean the folder beforehand so that I wouldn't accidentally upload something else? Or is there some filtering?
> But then I would need to clean the folder beforehand so that I wouldn't accidentally upload something else?

It will try to upload every backup source blob not already present in the backup remote, which has worked nicely in the past for users, do let us know if your use-case does not fit it so that we can learn about it.


You can work around this by instead of calling `conan source` directly in your second step, to run a `conan graph info --requires=exported/reference -c="&:tools.build:download_source=True"` call

This compues the dependency graph so it might not be quite optimal, but will force Conan to download the sources as if it were used compile the project, and those will properly by associated with your exported reference and can then be uploaded to the remote with the `conan upload` command.

Note that the reference is not associated in the `conan source` call as it does not compute enough information to ensure that it's always available, but maybe we can do something for some of the use-cases so that it references it where possible, will look into it, but no promises :)  
https://github.com/conan-io/conan/pull/18655 has been merged, closing this issue (using name/version from conanfile or arguments in the ``conan source`` command for the backup-sources feature)



Patch:
diff --git a/conan/internal/rest/download_cache.py b/conan/internal/rest/download_cache.py
index 9ee3a31227a..9d9c916fbdc 100644
--- a/conan/internal/rest/download_cache.py
+++ b/conan/internal/rest/download_cache.py
@@ -123,9 +123,17 @@ def update_backup_sources_json(cached_path, conanfile, urls):
         try:
             summary_key = str(conanfile.ref)
         except AttributeError:
-            # The recipe path would be different between machines
-            # So best we can do is to set this as unknown
-            summary_key = "unknown"
+            # If there's no node associated with the conanfile,
+            # try to construct a reference from the conanfile itself.
+            # We accept it if we have a name and a version at least.
+            if conanfile.name and conanfile.version:
+                user = f"@{conanfile.user}" if conanfile.user else ""
+                channel = f"/{conanfile.channel}" if conanfile.channel else ""
+                summary_key = f"{conanfile.name}/{conanfile.version}{user}{channel}"
+            else:
+                # The recipe path would be different between machines
+                # So best we can do is to set this as unknown
+                summary_key = "unknown"
 
         if not isinstance(urls, (list, tuple)):
             urls = [urls]


Test patch:
diff --git a/test/integration/cache/backup_sources_test.py b/test/integration/cache/backup_sources_test.py
index 344ce63d2f9..94e5184b97e 100644
--- a/test/integration/cache/backup_sources_test.py
+++ b/test/integration/cache/backup_sources_test.py
@@ -62,17 +62,16 @@ def source(self):
 
             assert 2 == len(os.listdir(os.path.join(tmp_folder, "s")))
             content = json.loads(load(os.path.join(tmp_folder, "s", sha256 + ".json")))
-            assert "http://localhost.mirror:5000/myfile.txt" in content["references"]["unknown"]
-            assert "http://localhost:5000/myfile.txt" in content["references"]["unknown"]
-            assert len(content["references"]["unknown"]) == 2
+            assert "http://localhost.mirror:5000/myfile.txt" in content["references"]["pkg/1.0"]
+            assert len(content["references"]["pkg/1.0"]) == 1
 
             # Ensure the cache is working and we didn't break anything by modifying the summary
             client.run("source .")
             assert "Downloading file" not in client.out
 
-            client.run("create .")
+            client.run("create . --user=barbarian")
             content = json.loads(load(os.path.join(tmp_folder, "s", sha256 + ".json")))
-            assert content["references"]["pkg/1.0"] == \
+            assert content["references"]["pkg/1.0@barbarian"] == \
                    ["http://localhost.mirror:5000/myfile.txt"]
 
             client.run("create . --user=barbarian --channel=stable")
@@ -214,9 +213,9 @@ def test_unknown_handling(self):
                             from conan.tools.files import download
                             class Pkg2(ConanFile):
                                 name = "pkg"
-                                version = "1.0"
                                 def source(self):
-                                    download(self, "{self.file_server.fake_url}/internet/myfile.txt", "myfile.txt",
+                                    download(self, "{self.file_server.fake_url}/internet/myfile.txt",
+                                             "myfile.txt",
                                              sha256="{sha256}")
                             """)
 
@@ -234,7 +233,7 @@ def source(self):
         s_folder = os.path.join(self.download_cache_folder, "s")
         assert len(os.listdir(s_folder)) == 2
 
-        self.client.run("export .")
+        self.client.run("export . --version=1.0")
         self.client.run("upload * -c -r=default")
         assert "No backup sources files to upload" in self.client.out
 
@@ -746,7 +745,8 @@ def source(self):
         # Ensure we are testing for an already uploaded recipe
         assert f"Recipe 'pkg/1.0#{exported_rev}' already in server, skipping upload" in self.client.out
 
-    def test_source_then_upload_workflow(self):
+    @pytest.mark.parametrize("unknown", [True, False])
+    def test_source_then_upload_workflow(self, unknown):
         mkdir(os.path.join(self.download_cache_folder, "s"))
 
         http_server_base_folder_internet = os.path.join(self.file_server.store, "internet")
@@ -770,7 +770,12 @@ def source(self):
                             f"core.sources:upload_url={self.file_server.fake_url}/backup/"})
 
         self.client.save({"conanfile.py": conanfile})
-        self.client.run("source .")
+        ref_args = "" if unknown else "--name foo --version 1.0"
+        reference_key = "unknown" if unknown else "foo/1.0"
+        self.client.run(f"source . {ref_args}")
+        content = json.loads(load(os.path.join(self.download_cache_folder, "s", sha256 + ".json")))
+        assert len(content["references"][reference_key]) == 1
+
         self.client.run("cache backup-upload")
         # This used to crash because we were trying to list a missing dir if only exports were made
         assert "[Errno 2] No such file or directory" not in self.client.out