Repository: conan-io/conan

Issue:
most requirements skipped with conan 1.5 when some are private
We have a conan setup that, when called with conan 1.4.5 or earlier, generates a visual studio property sheet with all our requirements. When called with 1.5, however, only one (Flow) of the requirements is included by the property sheet.

The conanfile.py that is used to generate the property sheet has a requirements field that looks like `requires = Flow.dependencies + ('Flow/'+Flow.Flow.version + "@velo3d/stable",)` where flow dependencies is a tuple like `dependencies = ('foo/1.2.3@velo3d/thirdparty',...` and the requires field on Flow looks like
```
    requires = dependencies + (
                ('X/b64300d87-5@velo3d/stable', 'private'),
                ('Y/d41fca23204-3@velo3d/stable', 'private'),
                ('Z/bcdf4b4f51a-2@velo3d/stable', 'private'),
                ('W/b155f94-1@velo3d/stable', 'private'),
                ('U/2d139f800c8-1@velo3d/stable', 'private'))
```

Going back to 1.4.5 or replacing Flow requires with
```
   requires = dependencies + (
                'X/b64300d87-5@velo3d/stable',
                'Y/d41fca23204-3@velo3d/stable',
                'Z/bcdf4b4f51a-2@velo3d/stable',
                'W/b155f94-1@velo3d/stable',
                'U/2d139f800c8-1@velo3d/stable')
```
fixes the problem.

Any ideas? Thanks.


Discussion:
Hi @drussel !

Yes, this seems a regression. 

I am having a look, but I don't fully understand the relation between the consumer conanfile and the ``Flow``? Is that another conanfile? It is a requirement? If you could please add some more details, we will have a look asap and try to release a fix.
Sorry, yes.

We have a `Flow.py` defines the Flow package and a `conanfile.py` that defines dependencies for the consumer project. The latter reuses the dependencies tuple from `Flow.py` to define its `self.requires` (to which it adds Flow itself as a requirement). The Flow package also has some private requirements that it appends to the dependencies tuple when defining its `self.requires`.

Is that clearer?
So, if I understood correctly, you want both the Flow package and the consumer conanfile.py package to have private requirements to all those packages. May I ask why those are private? It looks like you precisely want to make them public, so don't need the trick to use them from the dependency.

BTW, if you are in the CppLang slack team, please consider joining the #conan channel, it might be easier to discuss there.
In any case, I will start running some tests, it is not clear how to reproduce yet, but I will start checking. Thanks for the feedback!
Also, I am still missing how the consumer conanfile.py has access to ``Flow.dependencies``. The access to the ``Flow`` class is probably not a good idea, I wasn't aware myself that it was possible. How do you import it? 
The intention was only for the flow package to have private dependencies (they are only added to the requires field for that package). The conanfile.py that is used to build the property sheet doesn't have them directly (but is does depend on the flow package and a bunch of others that flow also depends). Although somehow they end up as interpreted as private in 1.5.

Since the conanfile.py is a python file (and doesn't apparently need to be stand alone), it can import other python files (`import Flow`) and take values from variables defined in them. This might not be intended usage of Conan and there may be better ways to solve the problem of keeping dependency versions in sync without going nuts :-) Overall, having some way of managing versions of packages independently from the list of package names would be great.

I'll try to throw together a simple reproducing case when I'm in the office tomorrow. Thanks for your time. Conan really solves a need we have.

Cool, I'll check out the Conan slack channel.
Really looking forward to getting your reproducible case, we will try to fix it as fast as possible.

> Overall, having some way of managing versions of packages independently from the list of package names would be great.

You mean, not having to specify versions in conanfiles? They would be global or something? Couldn't you use your current version ``self.version`` to define them?

See you in slack! Ask for the conan team there.

https://gist.github.com/drussel/7ef05d532cc71790a52bdd044bd15408 for MyPackage.py
https://gist.github.com/drussel/29fd8ff1364b9f2ca59d658a4f5ed403 for conanfile.py

command `python  -m conans.conan install --build=missing conanfile.py --update -s "compiler=Visual Studio" -s compiler.version=15 -s compiler.toolset=v140 -s build_type=Release -s compiler.runtime=MT -if .`

It appears to be fixed in 1.5.1.

Confirmed this is a regression, was able to reproduce. Will fix it asap and release in 1.5.2

Resolving patch:
diff --git a/conans/client/graph/graph.py b/conans/client/graph/graph.py
--- a/conans/client/graph/graph.py
+++ b/conans/client/graph/graph.py
@@ -47,6 +47,9 @@ def neighbors(self):
     def public_neighbors(self):
         return [edge.dst for edge in self.dependencies if not edge.private]
 
+    def private_neighbors(self):
+        return [edge.dst for edge in self.dependencies if edge.private]
+
     def inverse_neighbors(self):
         return [edge.src for edge in self.dependants]
 
diff --git a/conans/client/graph/graph_binaries.py b/conans/client/graph/graph_binaries.py
--- a/conans/client/graph/graph_binaries.py
+++ b/conans/client/graph/graph_binaries.py
@@ -130,19 +130,20 @@ def _evaluate_node(self, node, build_mode, update, evaluated_references, remote_
     def evaluate_graph(self, deps_graph, build_mode, update, remote_name):
         evaluated_references = {}
         for node in deps_graph.nodes:
-            conan_ref, conanfile = node.conan_ref, node.conanfile
-            if not conan_ref:
+            if not node.conan_ref:
                 continue
 
-            if [r for r in conanfile.requires.values() if r.private]:
+            private_neighbours = node.private_neighbors()
+            if private_neighbours:
                 self._evaluate_node(node, build_mode, update, evaluated_references, remote_name)
                 if node.binary != BINARY_BUILD:
-                    closure = deps_graph.full_closure(node)
-                    for node in closure:
-                        node.binary = BINARY_SKIP
+                    for neigh in private_neighbours:
+                        neigh.binary = BINARY_SKIP
+                        closure = deps_graph.full_closure(neigh)
+                        for n in closure:
+                            n.binary = BINARY_SKIP
 
         for node in deps_graph.nodes:
-            conan_ref, conanfile = node.conan_ref, node.conanfile
-            if not conan_ref or node.binary:
+            if not node.conan_ref or node.binary:
                 continue
             self._evaluate_node(node, build_mode, update, evaluated_references, remote_name)
diff --git a/conans/client/graph/printer.py b/conans/client/graph/printer.py
--- a/conans/client/graph/printer.py
+++ b/conans/client/graph/printer.py
@@ -1,21 +1,27 @@
 from conans.client.output import Color
 from conans.model.ref import PackageReference
 from conans.model.workspace import WORKSPACE_FILE
+from collections import OrderedDict
+from conans.client.graph.graph import BINARY_SKIP
 
 
 def print_graph(deps_graph, out):
-    all_nodes = []
-    ids = set()
-    for node in sorted(n for n in deps_graph.nodes if n.conan_ref):
+    requires = OrderedDict()
+    build_requires = OrderedDict()
+    for node in sorted(deps_graph.nodes):
+        if not node.conan_ref:
+            continue
         package_id = PackageReference(node.conan_ref, node.conanfile.info.package_id())
-        if package_id not in ids:
-            all_nodes.append(node)
-            ids.add(package_id)
-    requires = [n for n in all_nodes if not n.build_require]
+        if node.build_require:
+            build_requires.setdefault(package_id, []).append(node)
+        else:
+            requires.setdefault(package_id, []).append(node)
+
     out.writeln("Requirements", Color.BRIGHT_YELLOW)
 
     def _recipes(nodes):
-        for node in nodes:
+        for package_id, list_nodes in nodes.items():
+            node = list_nodes[0]  # For printing recipes, we can use the first one
             if node.remote == WORKSPACE_FILE:
                 from_text = "from '%s'" % WORKSPACE_FILE
             else:
@@ -26,13 +32,17 @@ def _recipes(nodes):
     out.writeln("Packages", Color.BRIGHT_YELLOW)
 
     def _packages(nodes):
-        for node in nodes:
-            ref, conanfile = node.conan_ref, node.conanfile
-            ref = PackageReference(ref, conanfile.info.package_id())
-            out.writeln("    %s - %s" % (repr(ref), node.binary), Color.BRIGHT_CYAN)
+        for package_id, list_nodes in nodes.items():
+            # The only way to have more than 1 states is to have 2
+            # and one is BINARY_SKIP (privates)
+            binary = set(n.binary for n in list_nodes)
+            if len(binary) > 1:
+                binary.remove(BINARY_SKIP)
+            assert len(binary) == 1
+            binary = binary.pop()
+            out.writeln("    %s - %s" % (repr(package_id), binary), Color.BRIGHT_CYAN)
     _packages(requires)
 
-    build_requires = [n for n in all_nodes if n.build_require]
     if build_requires:
         out.writeln("Build requirements", Color.BRIGHT_YELLOW)
         _recipes(build_requires)


Test patch:
diff --git a/conans/test/integration/private_deps_test.py b/conans/test/integration/private_deps_test.py
--- a/conans/test/integration/private_deps_test.py
+++ b/conans/test/integration/private_deps_test.py
@@ -1,7 +1,8 @@
+import os
 import unittest
+
 from conans.test.utils.tools import TestClient, TestServer
 from conans.model.ref import ConanFileReference
-import os
 from conans.test.utils.cpp_test_files import cpp_hello_conan_files
 from conans.paths import CONANINFO, BUILD_INFO_CMAKE
 from conans.util.files import load
@@ -9,6 +10,120 @@
 from nose.plugins.attrib import attr
 
 
+class PrivateBinariesTest(unittest.TestCase):
+
+    def test_private_skip(self):
+        client = TestClient()
+        conanfile = """from conans import ConanFile
+class Pkg(ConanFile):
+    def package_info(self):
+        self.cpp_info.libs = ["zlib"]
+"""
+        client.save({"conanfile.py": conanfile})
+        client.run("create . zlib/1.2.11@conan/stable")
+        client.save({"conanfile.py": conanfile.replace("zlib", "bzip2")})
+        client.run("create . bzip2/1.0.6@conan/stable")
+        conanfile = """from conans import ConanFile
+class MyPackage(ConanFile):
+    requires = ('zlib/1.2.11@conan/stable', ('bzip2/1.0.6@conan/stable', 'private'),)
+    def package_info(self):
+        self.cpp_info.libs = ["mypackage"]
+"""
+        client.save({"conanfile.py": conanfile})
+        client.run("create . MyPackage/1.0@testing/testing")
+        client.run("remove bzip2/1.0.6@conan/stable -p -f")
+        conanfile = """from conans import ConanFile
+class V3D(ConanFile):
+    requires = "zlib/1.2.11@conan/stable", "MyPackage/1.0@testing/testing"
+"""
+        client.save({"conanfile.py": conanfile})
+        client.run("install . -g=cmake")
+        self.assertIn("bzip2/1.0.6@conan/stable:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Skip",
+                      client.out)
+        self.assertIn("zlib/1.2.11@conan/stable:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache",
+                      client.out)
+        conanbuildinfo = load(os.path.join(client.current_folder, "conanbuildinfo.cmake"))
+        # The order is dictated by public MyPackage -> zlib dependency
+        self.assertIn("set(CONAN_LIBS mypackage zlib ${CONAN_LIBS})", conanbuildinfo)
+        self.assertNotIn("bzip2", conanbuildinfo)
+
+    def test_multiple_private_skip(self):
+        client = TestClient()
+        conanfile = """from conans import ConanFile
+class Pkg(ConanFile):
+    def package_info(self):
+        self.cpp_info.libs = ["zlib"]
+"""
+        client.save({"conanfile.py": conanfile})
+        client.run("create . zlib/1.2.11@conan/stable")
+        client.save({"conanfile.py": conanfile.replace("zlib", "bzip2")})
+        client.run("create . bzip2/1.0.6@conan/stable")
+        conanfile = """from conans import ConanFile
+class MyPackage(ConanFile):
+    requires = ('zlib/1.2.11@conan/stable', 'private'), ('bzip2/1.0.6@conan/stable', 'private')
+    def package_info(self):
+        self.cpp_info.libs = ["mypackage"]
+"""
+        client.save({"conanfile.py": conanfile})
+        client.run("create . MyPackage/1.0@testing/testing")
+        client.run("remove bzip2/1.0.6@conan/stable -p -f")
+        client.run("remove zlib* -p -f")
+        conanfile = """from conans import ConanFile
+class V3D(ConanFile):
+    requires = "zlib/1.2.11@conan/stable", "MyPackage/1.0@testing/testing"
+"""
+        client.save({"conanfile.py": conanfile})
+        error = client.run("install . -g=cmake", ignore_error=True)
+        self.assertTrue(error)
+        self.assertIn("ERROR: Missing prebuilt package for 'zlib/1.2.11@conan/stable'",
+                      client.out)
+        client.run("install zlib/1.2.11@conan/stable --build=missing")
+        client.run("install . -g=cmake")
+        self.assertIn("bzip2/1.0.6@conan/stable:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Skip",
+                      client.out)
+        self.assertIn("zlib/1.2.11@conan/stable:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache",
+                      client.out)
+        conanbuildinfo = load(os.path.join(client.current_folder, "conanbuildinfo.cmake"))
+        # The order is dictated by V3D declaration order, as other are privates
+        self.assertIn("set(CONAN_LIBS zlib mypackage ${CONAN_LIBS})", conanbuildinfo)
+        self.assertNotIn("bzip2", conanbuildinfo)
+
+    def test_own_private_skip(self):
+        client = TestClient()
+        conanfile = """from conans import ConanFile
+class Pkg(ConanFile):
+    def package_info(self):
+        self.cpp_info.libs = ["zlib"]
+"""
+        client.save({"conanfile.py": conanfile})
+        client.run("create . zlib/1.2.11@conan/stable")
+        client.save({"conanfile.py": conanfile.replace("zlib", "bzip2")})
+        client.run("create . bzip2/1.0.6@conan/stable")
+        conanfile = """from conans import ConanFile
+class MyPackage(ConanFile):
+    requires = 'zlib/1.2.11@conan/stable', ('bzip2/1.0.6@conan/stable', 'private')
+    def package_info(self):
+        self.cpp_info.libs = ["mypackage"]
+"""
+        client.save({"conanfile.py": conanfile})
+        client.run("create . MyPackage/1.0@testing/testing")
+        client.run("remove bzip2/1.0.6@conan/stable -p -f")
+        conanfile = """from conans import ConanFile
+class V3D(ConanFile):
+    requires = ("zlib/1.2.11@conan/stable", "private"), "MyPackage/1.0@testing/testing"
+"""
+        client.save({"conanfile.py": conanfile})
+        client.run("install . -g=cmake")
+        self.assertIn("bzip2/1.0.6@conan/stable:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Skip",
+                      client.out)
+        self.assertIn("zlib/1.2.11@conan/stable:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache",
+                      client.out)
+        conanbuildinfo = load(os.path.join(client.current_folder, "conanbuildinfo.cmake"))
+        # The order is dictated by public mypackage -> zlib
+        self.assertIn("set(CONAN_LIBS mypackage zlib ${CONAN_LIBS})", conanbuildinfo)
+        self.assertNotIn("bzip2", conanbuildinfo)
+
+
 @attr("slow")
 class PrivateDepsTest(unittest.TestCase):