Repository: conan-io/conan

Issue:
conan info generates bad .conan_link files
To help us debug your issue please explain:

- [x] I've read the [CONTRIBUTING guide](https://github.com/conan-io/conan/blob/develop/.github/CONTRIBUTING.md).
- [x] I've specified the Conan version, operating system version and any tool that can be relevant.
- [x] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

Operating System: Windows 10 64-bit
Conan version: v1.11.2

I have a situation where I have two recipes - A and B, and recipe A depends on recipe B. What I do to determine the depdencies between my recipes is:
1. Run `conan export` over all recipes
2. Run `conan info -g out.dot` to generate the dependency graph for all recipes
3. Parse the dependency graph for each recipe to generate the full dependency graph

Once I have the complete graph, I can then traverse it to build the recipes in the right order.

The issue I have is when I run conan info on recipe A (which depends on B), on Windows, I get a `c:\Users\MyUser\.conan\data\recipeB\0.0.2\user\channel\package\69f10d1e6d78c1962fbe8fb80440f32b1fdf8423\.conan_link`.

And that `.conan_link` points to a directory which doesn't exist. So if I now tried to do an upload of all recipeB packages, I get an error from conan saying that particular package is corrupted. Removing that particular package folder (the one with a hash as its name) fixes the issue.

Do you think this issue can be fixed in conan? Currently, we work around the issue by specifically looking for these broken `.conan_link` files and removing the package folders they're in.


Discussion:
Sorry, another thing I should have mentioned is that recipeB in the above example has `short_paths` set to `True`, which is what causes the `.conan_link` file to be generated.
Hi @hebaishi 

I am having a look. In the meantime I'd like to ask, have you had a look to the ``conan info --build-order`` functionality? That gives you the information of the build order, already in parallelizable groups.
Yes, I have had a look at that, but the problem is there is information in the graph which is missing in the build order list. And as we are doing this for ~70 recipes, it's important we build things in the right order, so the build graph works better for our purposes.

The workaround for this issue isn't actually too bad, and because the approach we now use (based on your recommendation) uses conan to get the dependencies between the recipes, it's actually incredibly robust. Fixing this issue would just be the cherry on top of what is already a pretty amazing pie. :+1:

Thank you again for suggesting `conan export`!

Resolving patch:
diff --git a/conans/client/cmd/uploader.py b/conans/client/cmd/uploader.py
--- a/conans/client/cmd/uploader.py
+++ b/conans/client/cmd/uploader.py
@@ -136,10 +136,14 @@ def _collect_packages_to_upload(self, refs, confirm, remote_name, all_packages,
                     raise NotFoundException(("There is no local conanfile exported as %s" %
                                              str(ref)))
 
-                package_layout = self._cache.package_layout(ref)
-                if all_packages:
-                    packages_ids = package_layout.conan_packages()
-                elif query:
+                # TODO: This search of binary packages has to be improved, more robust
+                # So only real packages are retrieved
+                if all_packages or query:
+                    if all_packages:
+                        query = None
+                    # better to do a search, that will retrieve real packages with ConanInfo
+                    # Not only "package_id" folders that could be empty
+                    package_layout = self._cache.package_layout(ref.copy_clear_rev())
                     packages = search_packages(package_layout, query)
                     packages_ids = list(packages.keys())
                 elif package_id:
@@ -164,7 +168,11 @@ def _collect_packages_to_upload(self, refs, confirm, remote_name, all_packages,
                 prefs = []
                 # Gather all the complete PREFS with PREV
                 for package_id in packages_ids:
+                    if package_id not in metadata.packages:
+                        raise ConanException("Binary package %s:%s not found"
+                                             % (str(ref), package_id))
                     package_revision = metadata.packages[package_id].revision
+                    assert package_revision is not None, "PREV cannot be None to upload"
                     prefs.append(PackageReference(ref, package_id, package_revision))
                 refs_by_remote[remote].append((ref, conanfile, prefs))
 
diff --git a/conans/client/remover.py b/conans/client/remover.py
--- a/conans/client/remover.py
+++ b/conans/client/remover.py
@@ -51,7 +51,8 @@ def remove_builds(self, package_layout, ids=None):
         if not ids:
             path = package_layout.builds()
             for build in package_layout.conan_builds():
-                self._remove(os.path.join(path, build), package_layout.ref, "build folder:%s" % build)
+                self._remove(os.path.join(path, build), package_layout.ref,
+                             "build folder:%s" % build)
             self._remove(path, package_layout.ref, "builds")
         else:
             for id_ in ids:


Test patch:
diff --git a/conans/test/functional/command/info_folders_test.py b/conans/test/functional/command/info_folders_test.py
--- a/conans/test/functional/command/info_folders_test.py
+++ b/conans/test/functional/command/info_folders_test.py
@@ -3,12 +3,15 @@
 import re
 import subprocess
 import unittest
+from textwrap import dedent
 
 from conans.client import tools
 from conans.model.ref import ConanFileReference, PackageReference
 from conans.paths import CONANFILE
 from conans.test.utils.test_files import temp_folder
-from conans.test.utils.tools import NO_SETTINGS_PACKAGE_ID, TestClient
+from conans.test.utils.tools import NO_SETTINGS_PACKAGE_ID, TestClient,\
+    TestServer
+
 
 conanfile_py = """
 from conans import ConanFile
@@ -108,9 +111,8 @@ def test_single_field(self):
         self.assertIn(os.path.join(base_path, "build"), output)
         self.assertNotIn("package", output)
 
+    @unittest.skipIf(platform.system() != "Windows", "Needs windows for short_paths")
     def test_short_paths(self):
-        if platform.system() != "Windows":
-            return
         folder = temp_folder(False)
         short_folder = os.path.join(folder, ".cn")
 
@@ -141,23 +143,23 @@ def test_short_paths(self):
                 self.assertFalse(os.path.exists(path))
                 self.assertTrue(os.path.exists(os.path.dirname(path)))
 
+    @unittest.skipIf(platform.system() != "Windows", "Needs windows for short_paths")
     def test_short_paths_home_set_acl(self):
         """
         When CONAN_USER_HOME_SHORT is living in NTFS file systems, current user needs to be
-        granted with full control permission to avoid access problems when cygwin/msys2 windows subsystems
-        are mounting/using that folder.
+        granted with full control permission to avoid access problems when cygwin/msys2
+        windows subsystems are mounting/using that folder.
         """
-        if platform.system() != "Windows":
-            return
-
         folder = temp_folder(False)  # Creates a temporary folder in %HOME%\appdata\local\temp
 
-        out = subprocess.check_output("wmic logicaldisk %s get FileSystem" % os.path.splitdrive(folder)[0])
+        out = subprocess.check_output("wmic logicaldisk %s get FileSystem"
+                                      % os.path.splitdrive(folder)[0])
         if "NTFS" not in str(out):
             return
         short_folder = os.path.join(folder, ".cnacls")
 
-        self.assertFalse(os.path.exists(short_folder), "short_folder: %s shouldn't exists" % short_folder)
+        self.assertFalse(os.path.exists(short_folder), "short_folder: %s shouldn't exists"
+                         % short_folder)
         os.makedirs(short_folder)
 
         current_domain = os.environ['USERDOMAIN']
@@ -179,7 +181,8 @@ def test_short_paths_home_set_acl(self):
 
         # Retrieve ACLs from short_folder
         try:
-            short_folder_acls = subprocess.check_output("cacls %s" % short_folder, stderr=subprocess.STDOUT)
+            short_folder_acls = subprocess.check_output("cacls %s" % short_folder,
+                                                        stderr=subprocess.STDOUT)
         except subprocess.CalledProcessError as e:
             raise Exception("Error %s getting ACL from short_folder: '%s'." % (e, short_folder))
 
@@ -196,3 +199,29 @@ def test_direct_conanfile(self):
         self.assertNotIn("source_folder", output)
         self.assertNotIn("build_folder", output)
         self.assertNotIn("package_folder", output)
+
+    @unittest.skipIf(platform.system() != "Windows", "Needs windows for short_paths")
+    def test_short_paths_folders(self):
+        # https://github.com/conan-io/conan/issues/4612
+        folder = temp_folder(False)
+        short_folder = os.path.join(folder, ".cn")
+
+        conanfile = dedent("""
+            from conans import ConanFile
+
+            class Conan(ConanFile):
+                short_paths=True
+            """)
+        with tools.environment_append({"CONAN_USER_HOME_SHORT": short_folder}):
+            client = TestClient(base_folder=folder, servers={"default": TestServer()},
+                                users={"default": [("lasote", "mypass")]})
+            client.save({CONANFILE: conanfile})
+            client.run("export . pkga/0.1@lasote/testing")
+            client.run("info pkga/0.1@lasote/testing --paths")
+            self.assertIn("source_folder: %s" % short_folder, client.out)
+            self.assertIn("build_folder: %s" % short_folder, client.out)
+            self.assertIn("package_folder: %s" % short_folder, client.out)
+
+            client.run("upload pkga/0.1@lasote/testing --all")
+            self.assertIn("Uploaded conan recipe", client.out)
+            self.assertNotIn("Uploading package 1/1", client.out)
diff --git a/conans/test/functional/command/upload_test.py b/conans/test/functional/command/upload_test.py
--- a/conans/test/functional/command/upload_test.py
+++ b/conans/test/functional/command/upload_test.py
@@ -12,6 +12,7 @@
 from conans.test.utils.cpp_test_files import cpp_hello_conan_files
 from conans.test.utils.tools import NO_SETTINGS_PACKAGE_ID, TestClient, TestServer
 from conans.util.files import gzopen_without_timestamps, is_dirty, save
+from conans.test.utils.conanfile import TestConanFile
 
 conanfile = """from conans import ConanFile
 class MyPkg(ConanFile):
@@ -37,6 +38,14 @@ def package(self):
 
 class UploadTest(unittest.TestCase):
 
+    def test_upload_not_existing(self):
+        client = TestClient(servers={"default": TestServer()},
+                            users={"default": [("lasote", "mypass")]})
+        client.save({"conanfile.py": str(TestConanFile("Hello", "0.1"))})
+        client.run("export . Hello/0.1@lasote/testing")
+        client.run("upload Hello/0.1@lasote/testing -p=123", assert_error=True)
+        self.assertIn("ERROR: Binary package Hello/0.1@lasote/testing:123 not found", client.out)
+
     def not_existing_error_test(self):
         """ Trying to upload with pattern not matched must raise an Error
         """
diff --git a/conans/test/functional/old/package_integrity_test.py b/conans/test/functional/old/package_integrity_test.py
--- a/conans/test/functional/old/package_integrity_test.py
+++ b/conans/test/functional/old/package_integrity_test.py
@@ -3,8 +3,9 @@
 
 from conans.model.ref import ConanFileReference, PackageReference
 from conans.test.utils.conanfile import TestConanFile
-from conans.test.utils.tools import TestClient, TestServer
-from conans.util.files import save, set_dirty
+from conans.test.utils.tools import TestClient, TestServer,\
+    NO_SETTINGS_PACKAGE_ID
+from conans.util.files import set_dirty
 
 
 class PackageIngrityTest(unittest.TestCase):
@@ -33,21 +34,17 @@ def upload_dirty_test(self):
         client = TestClient(servers={"default": test_server},
                             users={"default": [("lasote", "mypass")]})
         client.save({"conanfile.py": str(TestConanFile())})
-        client.run("export . lasote/testing")
+        client.run("create . lasote/testing")
         ref = ConanFileReference.loads("Hello/0.1@lasote/testing")
-        pref = PackageReference(ref, "12345")
+        pref = PackageReference(ref, NO_SETTINGS_PACKAGE_ID)
         package_folder = client.cache.package(pref)
-        recipe_rev = client.cache.package_layout(ref).recipe_revision()
-        with client.cache.package_layout(pref.ref).update_metadata() as metadata:
-            metadata.packages[pref.id].revision = "Fakerevision"
-            metadata.packages[pref.id].recipe_revision = recipe_rev
-        save(os.path.join(package_folder, "conanmanifest.txt"), "888")
         set_dirty(package_folder)
 
         client.run("upload * --all --confirm", assert_error=True)
-        self.assertIn("ERROR: Package Hello/0.1@lasote/testing:12345 is corrupted, aborting upload",
+        self.assertIn("ERROR: Package %s is corrupted, aborting upload" % str(pref),
                       client.out)
-        self.assertIn("Remove it with 'conan remove Hello/0.1@lasote/testing -p=12345'", client.out)
+        self.assertIn("Remove it with 'conan remove Hello/0.1@lasote/testing -p=%s'"
+                      % NO_SETTINGS_PACKAGE_ID, client.out)
 
-        client.run("remove Hello/0.1@lasote/testing -p=12345 -f")
+        client.run("remove Hello/0.1@lasote/testing -p=%s -f" % NO_SETTINGS_PACKAGE_ID)
         client.run("upload * --all --confirm")