Repository: conan-io/conan

Issue:
[bug] requires of a dependency of a dependency in build context not visible
### Describe the bug

conan 2.3.0

For building Matlab Code with [s-functions](https://www.mathworks.com/help/simulink/sfg/what-is-an-s-function.html), certain packages have to be propagated with `build=True, visible=True` traits. It seems however that the dependencies of such packages in build context disappear at some point.

As minimal example, we have the following dependency tree:
`pkgb` -> `libpkgb` -> `libpkgb_dependency`

where `pkgb` contains the sfunction which should only reside in build context (think about a cross build scenario, the sfunction has to execute in Matlab only at build time and generates code), this sfunction depends on `libpkgb` (some shared library)  and the generated code also depends on `libpgkb` so that one has to be available in both contexts. `libpkgb_dependency` (another shared library) is just a dependency of `libpkgb`, so it also has to be available in both contexts.

This s-function is then consumed by `pkga` (e.g. a Simulink model) as follows:
`pkga` (host context) -> `libpkgb`
`pkga` (build context) -> `pkgb` with `visible=True` trait.

It now seems that the dependencies of `pkgb` in build context disappear when `pkga` is consumed by `pkga_consumer` in host context. This makes it impossible to e.g. aggregate Matlab packages with s-functions for opening Matlab inside a `VirtualBuildEnv` to e.g. generate code.

Please see below zip file for exemplary conanfiles and the requirement traits used (maybe those are also incomplete?).

### How to reproduce it

Execute `./create_all.sh`:
`pkga` has `libpkgb` in its build requirements as expected:
```
pkga/sth@sth/sth: Forced build from source
Requirements
    libpkgb/sth@sth/sth#86fc06aaf2beb0ade77891530e88ba05:3e72f5b11d4ddf4d622f2dba22a387dac8e39b7e#572353169b3c10509f45336eef0f606a - Cache
    libpkgb_dependency/sth@sth/sth#95e394580d8db0d9b4bf47d71722d0e6:da39a3ee5e6b4b0d3255bfef95601890afd80709#0ba8627bd47edc3a501e8f0eb9a79e5e - Cache
    pkga/sth@sth/sth#5de6164f0525938b5251b51dce86bb93:a9fea83b8cb8528171eec79a036ec70971964fe6 - Build
Build requirements
    libpkgb/sth@sth/sth#86fc06aaf2beb0ade77891530e88ba05:3e72f5b11d4ddf4d622f2dba22a387dac8e39b7e#572353169b3c10509f45336eef0f606a - Cache
    libpkgb_dependency/sth@sth/sth#95e394580d8db0d9b4bf47d71722d0e6:da39a3ee5e6b4b0d3255bfef95601890afd80709#0ba8627bd47edc3a501e8f0eb9a79e5e - Cache
    pkgb/sth@sth/sth#64a4a8d5db953083748e775d6a8e6ac2:a9fea83b8cb8528171eec79a036ec70971964fe6#f0d11924c47218dedcea613e83bcfcff - Cache
```

`pkga_consumer` has `libpkgb` not anymore in its build requirements:
```
pkga_consumer/sth@sth/sth: Forced build from source
Requirements
    libpkgb/sth@sth/sth#86fc06aaf2beb0ade77891530e88ba05:3e72f5b11d4ddf4d622f2dba22a387dac8e39b7e#572353169b3c10509f45336eef0f606a - Cache
    libpkgb_dependency/sth@sth/sth#95e394580d8db0d9b4bf47d71722d0e6:da39a3ee5e6b4b0d3255bfef95601890afd80709#0ba8627bd47edc3a501e8f0eb9a79e5e - Cache
    pkga/sth@sth/sth#5de6164f0525938b5251b51dce86bb93:a9fea83b8cb8528171eec79a036ec70971964fe6#f0d11924c47218dedcea613e83bcfcff - Cache
    pkga_consumer/sth@sth/sth#36e174720d405f5bf14b41af15132a1e:163876653e2bf77f8d6c9a484c16abf689d068ed - Build
Build requirements
    pkgb/sth@sth/sth#64a4a8d5db953083748e775d6a8e6ac2:a9fea83b8cb8528171eec79a036ec70971964fe6#f0d11924c47218dedcea613e83bcfcff - Cache
Skipped binaries
    libpkgb/sth@sth/sth, libpkgb_dependency/sth@sth/sth
```
[dependency_propagation_build.zip](https://github.com/conan-io/conan/files/15420052/dependency_propagation_build.zip)



Discussion:
Hi @maximilianmuehlbauer 

Thanks for your report.

To summarize the graph would be:

```
pkga_consumer -(requires)-> pkga(shared) --------(requires)--------->libb(shared) -(requires)-> libb_dep(shared)
                                    \--(build-require) ->pkgb(shared)-->/
```

I'd like to start with a recommendation: ``build=True, visible=True`` will be really discouraged in most cases. I have already had some experience trying it, and I would advice against this.

If the intention is that the ``pkga_consumer`` needs some ``tool-requires`` for itself, because it needs such tool requires to build itself, then that means that ``pkgb`` (shared-library) should be a direct dependency for ``pkga_consumer`` and it shouldn't be inherited from ``pkga``

It would be also good to understand how shared libraries can be used as "tool-requires" (``build=True``), as this puts such libraries in the "build" context, so the binaries might actually be incompatible (cross-compilation) or inconsistent/incorrect (building for debug will still bring Release for the build context by default).

Thanks very much for your reproducible code, I will use it to make a test, and understand what could be happening.


Thanks for the quick reply!

> I'd like to start with a recommendation: `build=True, visible=True` will be really discouraged in most cases. I have already had some experience trying it, and I would advice against this.

Yes, Matlab is a very special case... one might have a model that depends on a Simulink library (like a C++ header-only lib), that depends on a library, ... and at some point there's an sfunction. For building the final model, all of that is needed in the build environment which is where we run Matlab to generate the C++ code.

> If the intention is that the `pkga_consumer` needs some `tool-requires` for itself, because it needs such tool requires to build itself, then that means that `pkgb` (shared-library) should be a direct dependency for `pkga_consumer` and it shouldn't be inherited from `pkga`

Yes, essentially that is what we want; however making an explicit tool requirement from `pkga_consumer` to the sfunction is hard as the name, version etc. of that sfunction would need to stay in sync with a possibly long dependency tree in between. We would like to give Matlab developers the possibility to depend on a library which might come with such sfunction in some place without explicitly having to specify that sfunction.

> It would be also good to understand how shared libraries can be used as "tool-requires" (`build=True`), as this puts such libraries in the "build" context, so the binaries might actually be incompatible (cross-compilation) or inconsistent/incorrect (building for debug will still bring Release for the build context by default).

Yes, those libraries will be incompatible (and cross-compiling actually works with `pkga` as root). The sfunction in `pkgb` might need `libpkgb` to generate code (but it's a normal requires of `pkgb` -> to my understanding, it should land in build context as soon as `pkg` is put in build context in that case) while the generated code links against `libpkgb` for possibly a different architecture.
I have been having a look and so far it seems the ``build=True`` trait has higher priority than the ``visible=True`` one, so build-requires are still not being fully propagated. I will keep investigating tomorrow if it is possible to do this propagation without risks of breaking.
This is quite challenging, and needs to be considered (small) step by step, defining how build requirements need to propagate when made visible requires some deeper research and careful consider what could be breaking. I am assigning it for 2.5 (2.4 will be soon), to start working on it, but it might take longer time and require a few iterations and releases to move forward.
> This is quite challenging, and needs to be considered (small) step by step, defining how build requirements need to propagate when made visible requires some deeper research and careful consider what could be breaking. I am assigning it for 2.5 (2.4 will be soon), to start working on it, but it might take longer time and require a few iterations and releases to move forward.

Thanks for the feedback!  We're happy to test with our use case as soon as that's possible. From having a quick look, could it be that this issues comes from the following lines where `visible=False` is being set: https://github.com/conan-io/conan/blob/36148df880d137e88897bb429e752186be58f313/conans/model/requires.py#L280-L281



Patch:
diff --git a/conans/model/requires.py b/conans/model/requires.py
index 23987d0dc81..b1974a7dc6f 100644
--- a/conans/model/requires.py
+++ b/conans/model/requires.py
@@ -270,15 +270,18 @@ def transform_downstream(self, pkg_type, require, dep_pkg_type):
 
         if require.build:  # public!
             # TODO: To discuss if this way of conflicting build_requires is actually useful or not
+            # Build-requires will propagate its main trait for running exes/shared to downstream
+            # consumers so run=require.run, irrespective of the 'self.run' trait
             downstream_require = Requirement(require.ref, headers=False, libs=False, build=True,
-                                             run=False, visible=self.visible, direct=False)
+                                             run=require.run, visible=self.visible, direct=False)
             return downstream_require
 
         if self.build:  # Build-requires
             # If the above is shared or the requirement is explicit run=True
+            # visible=self.visible will further propagate it downstream
             if dep_pkg_type is PackageType.SHARED or require.run:
                 downstream_require = Requirement(require.ref, headers=False, libs=False, build=True,
-                                                 run=True, visible=False, direct=False)
+                                                 run=True, visible=self.visible, direct=False)
                 return downstream_require
             return
 


Test patch:
diff --git a/test/integration/graph/core/test_build_requires.py b/test/integration/graph/core/test_build_requires.py
index acf8c40e23b..7e396044b72 100644
--- a/test/integration/graph/core/test_build_requires.py
+++ b/test/integration/graph/core/test_build_requires.py
@@ -639,7 +639,89 @@ def test_simple(self):
         # node, include, link, build, run
         _check_transitive(lib, [(cmake, False, False, True, True)])
         _check_transitive(app, [(lib, True, True, False, False),
-                                (cmake, False, False, True, False)])
+                                (cmake, False, False, True, True)])
+
+    def test_deep_dependency_tree(self):
+        # app -> liba -> libb-(br public) -> sfun -> libsfun -> libx -> liby -> libz
+        #                    -(normal req) -> libsfun -> libx -> liby -> libz
+        self.recipe_conanfile("libz/0.1", GenConanfile())
+        self.recipe_conanfile("liby/0.1", GenConanfile().with_requirement("libz/0.1", run=True))
+        self.recipe_conanfile("libx/0.1", GenConanfile().with_requirement("liby/0.1", run=True))
+        self.recipe_conanfile("libsfun/0.1", GenConanfile().with_requirement("libx/0.1", run=True))
+        self.recipe_conanfile("sfun/0.1", GenConanfile().with_requirement("libsfun/0.1", run=True))
+        self.recipe_conanfile("libb/0.1", GenConanfile()
+                              .with_tool_requirement("sfun/0.1", visible=True)
+                              .with_requirement("libsfun/0.1", run=True))
+        self.recipe_conanfile("liba/0.1", GenConanfile().with_requirement("libb/0.1", run=True))
+        deps_graph = self.build_graph(GenConanfile("app", "0.1").with_requirement("liba/0.1", run=True))
+
+        # Build requires always apply to the consumer
+        self.assertEqual(8 + 4, len(deps_graph.nodes))
+        app = deps_graph.root
+        liba = app.dependencies[0].dst
+        libb = liba.dependencies[0].dst
+        libsfun = libb.dependencies[0].dst
+        libx = libsfun.dependencies[0].dst
+        liby = libx.dependencies[0].dst
+        libz = liby.dependencies[0].dst
+        sfun = libb.dependencies[1].dst
+        libsfun_build = sfun.dependencies[0].dst
+        libx_build = libsfun_build.dependencies[0].dst
+        liby_build = libx_build.dependencies[0].dst
+        libz_build = liby_build.dependencies[0].dst
+
+        # TODO non-build-requires
+
+        self._check_node(app, "app/0.1@", deps=[liba], dependents=[])
+        self._check_node(liba, "liba/0.1#123", deps=[libb], dependents=[app])
+        self._check_node(libb, "libb/0.1#123", deps=[sfun, libsfun], dependents=[liba])
+        self._check_node(sfun, "sfun/0.1#123", deps=[libsfun_build], dependents=[libb])
+        self._check_node(libsfun_build, "libsfun/0.1#123", deps=[libx_build], dependents=[sfun])
+        self._check_node(libx_build, "libx/0.1#123", deps=[liby_build], dependents=[libsfun_build])
+        self._check_node(liby_build, "liby/0.1#123", deps=[libz_build], dependents=[libx_build])
+        self._check_node(libz_build, "libz/0.1#123", deps=[], dependents=[liby_build])
+
+        # node, include, link, build, run
+        _check_transitive(liby_build, [(libz_build, True, True, False, True)])
+        _check_transitive(libx_build, [(liby_build, True, True, False, True),
+                                       (libz_build, True, True, False, True)])
+        _check_transitive(libsfun_build, [(libx_build, True, True, False, True),
+                                          (liby_build, True, True, False, True),
+                                          (libz_build, True, True, False, True)])
+        _check_transitive(sfun, [(libsfun_build, True, True, False, True),
+                                 (libx_build, True, True, False, True),
+                                 (liby_build, True, True, False, True),
+                                 (libz_build, True, True, False, True)])
+        _check_transitive(libb, [(libsfun, True, True, False, True),
+                                 (libx, True, True, False, True),
+                                 (liby, True, True, False, True),
+                                 (libz, True, True, False, True),
+                                 (sfun, False, False, True, True),
+                                 (libsfun_build, False, False, True, True),
+                                 (libx_build, False, False, True, True),
+                                 (liby_build, False, False, True, True),
+                                 (libz_build, False, False, True, True)])
+        _check_transitive(liba, [(libb, True, True, False, True),
+                                 (libsfun, True, True, False, True),
+                                 (libx, True, True, False, True),
+                                 (liby, True, True, False, True),
+                                 (libz, True, True, False, True),
+                                 (sfun, False, False, True, True),
+                                 (libsfun_build, False, False, True, True),
+                                 (libx_build, False, False, True, True),
+                                 (liby_build, False, False, True, True),
+                                 (libz_build, False, False, True, True)])
+        _check_transitive(app, [(liba, True, True, False, True),
+                                (libb, True, True, False, True),
+                                (libsfun, True, True, False, True),
+                                (libx, True, True, False, True),
+                                (liby, True, True, False, True),
+                                (libz, True, True, False, True),
+                                (sfun, False, False, True, True),
+                                (libsfun_build, False, False, True, True),
+                                (libx_build, False, False, True, True),
+                                (liby_build, False, False, True, True),
+                                (libz_build, False, False, True, True)])
 
     def test_conflict_diamond(self):
         # app -> libb -(br public)-> cmake/0.1
@@ -668,6 +750,40 @@ def test_conflict_diamond(self):
         self._check_node(libb, "libb/0.1#123", deps=[cmake1], dependents=[app])
         self._check_node(cmake1, "cmake/0.1#123", deps=[], dependents=[libb])
 
+    def test_conflict_diamond_two_levels(self):
+        # app -> libd -> libb -(br public)-> cmake/0.1
+        #   \--> libe -> libc -(br public)-> cmake/0.2
+        self.recipe_conanfile("cmake/0.1", GenConanfile())
+        self.recipe_conanfile("cmake/0.2", GenConanfile())
+        self.recipe_conanfile("libb/0.1",
+                              GenConanfile().with_tool_requirement("cmake/0.1", visible=True))
+        self.recipe_conanfile("libc/0.1",
+                              GenConanfile().with_tool_requirement("cmake/0.2", visible=True))
+        self.recipe_conanfile("libd/0.1", GenConanfile().with_requires("libb/0.1"))
+        self.recipe_conanfile("libe/0.1", GenConanfile().with_requires("libc/0.1"))
+
+        deps_graph = self.build_graph(GenConanfile("app", "0.1").with_requires("libd/0.1",
+                                                                               "libe/0.1"),
+                                      install=False)
+
+        assert type(deps_graph.error) == GraphConflictError
+
+        # Build requires always apply to the consumer
+        self.assertEqual(6, len(deps_graph.nodes))
+        app = deps_graph.root
+        libd = app.dependencies[0].dst
+        libe = app.dependencies[1].dst
+        libb = libd.dependencies[0].dst
+        libc = libe.dependencies[0].dst
+        cmake1 = libb.dependencies[0].dst
+
+        self._check_node(app, "app/0.1@", deps=[libd, libe], dependents=[])
+        self._check_node(libd, "libd/0.1#123", deps=[libb], dependents=[app])
+        self._check_node(libe, "libe/0.1#123", deps=[libc], dependents=[app])
+        self._check_node(libb, "libb/0.1#123", deps=[cmake1], dependents=[libd])
+        self._check_node(libc, "libc/0.1#123", deps=[], dependents=[libe])
+        self._check_node(cmake1, "cmake/0.1#123", deps=[], dependents=[libb])
+
     def test_tool_requires(self):
         # app -> libb -(br public)-> protobuf/0.1
         #           \--------------> protobuf/0.2
@@ -695,7 +811,7 @@ def test_tool_requires(self):
         # node, headers, lib, build, run
         _check_transitive(app, [(libb, True, True, False, False),
                                 (protobuf_host, True, True, False, False),
-                                (protobuf_build, False, False, True, False)])
+                                (protobuf_build, False, False, True, True)])
 
     def test_tool_requires_override(self):
         # app -> libb -(br public)-> protobuf/0.1
@@ -726,7 +842,7 @@ def test_tool_requires_override(self):
         # node, headers, lib, build, run
         _check_transitive(app, [(libb, True, True, False, False),
                                 (protobuf_host, True, True, False, False),
-                                (protobuf_build, False, False, True, False)])
+                                (protobuf_build, False, False, True, True)])
         _check_transitive(libb, [(protobuf_host, True, True, False, False),
                                  (protobuf_build, False, False, True, True)])