Repository: conan-io/conan

Issue:
[bug] Dependencies' build_policy overwrites command line parameter
We recently encountered an issue with our CI system, when incorrectly configured dependency packages were rebuilt in a CI step, where we didn't allow it via `--build=never`.

### Environment Details
  * Operating System+version: Windows 10
  * Conan version: 1.22.0 and 1.23.0
  * Python version: 3.6.4

### Steps to reproduce

a/conanfile.py
```python
from conans import ConanFile

class TestA(ConanFile):
  name = "TestA"
  version = "1.0"
  requires = "TestB/1.0@test/test"

  def build(self):
    self.output.info("Build TestA")
```

b/conanfile.py
```python
from conans import ConanFile

class TestB(ConanFile):
  name = "TestB"
  version = "1.0"
  build_policy = "missing" # !!!

  def build(self):
    self.output.info("Build TestB")
```

1. Create packages TestA and TestB
   ```
   conan create b test/test
   conan create a test/test
   ```
2. Remove package TestB and reexport the conanfile
   ```
   conan remove TestB
   conan export b test/test
   ```
3. Install TestA without build
   ```
   conan install TestA/1.0@test/test --build=never
   ```

Expected behavior:

Install command should fail, since TestB is not installed and `--build=never` is specified.

Actual behavior:

Install command succeeds and TestB is rebuilt (see log).

Remark: The same bug occurs when setting `build_policy=always` in `b/conanfile.py`. You can then skip step 2 and specify `--build=missing` in step 3, then TestB will be rebuilt although it shouldn't.

### Logs

Log for step 3:
```
PS E:\Workspace\conan_bug> conan install TestA/1.0@test/test --build=never
Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=Visual Studio
compiler.runtime=MD
compiler.version=15
os=Windows
os_build=Windows
[options]
[build_requires]
[env]

TestB/1.0@test/test: Building package from source as defined by build_policy='missing'
Installing package: TestA/1.0@test/test
Requirements
    TestA/1.0@test/test from local cache - Cache
    TestB/1.0@test/test from local cache - Cache
Packages
    TestA/1.0@test/test:5c07eea89709163d6f80759ede846381d3745442 - Cache
    TestB/1.0@test/test:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Build

Installing (downloading, building) binaries...
TestB/1.0@test/test: Configuring sources in C:\Users\Oliver\.conan\data\TestB\1.0\test\test\source
TestB/1.0@test/test: Copying sources to build folder
TestB/1.0@test/test: Building your package in C:\Users\Oliver\.conan\data\TestB\1.0\test\test\build\5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9
TestB/1.0@test/test: Generator txt created conanbuildinfo.txt
TestB/1.0@test/test: Calling build()
TestB/1.0@test/test: Build TestB
TestB/1.0@test/test: Package '5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9' built
TestB/1.0@test/test: Build folder C:\Users\Oliver\.conan\data\TestB\1.0\test\test\build\5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9
TestB/1.0@test/test: Generated conaninfo.txt
TestB/1.0@test/test: Generated conanbuildinfo.txt
TestB/1.0@test/test: Generating the package
TestB/1.0@test/test: Package folder C:\Users\Oliver\.conan\data\TestB\1.0\test\test\package\5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9
TestB/1.0@test/test: Calling package()
TestB/1.0@test/test: WARN: This conanfile has no package step
TestB/1.0@test/test package(): WARN: No files in this package!
TestB/1.0@test/test: Package '5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9' created
TestB/1.0@test/test: Created package revision 4d0ed6b951018d8af295b25c8922813c
TestA/1.0@test/test: Already installed!
```


Discussion:
Hi @FalkorX 

Thanks for your detailed report.

This behavior is by design, not a bug. The ``build_policy`` on recipes was a mechanism requested by users to allow some recipes to build from sources irrespective of the ``--build`` argument of the command line. The default behavior of ``conan install`` is already never to build packages. This is documented in the docs, and in the command line help (see last lines):

```
-b [BUILD], --build [BUILD]
                            Optional, use it to choose if you want to build from
                            sources: --build Build all from sources, do not use
                            binary packages. --build=never Never build, use binary
                            packages or fail if a binary package is not found.
                            --build=missing Build from code if a binary package is
                            not found. --build=cascade Will build from code all
                            the nodes with some dependency being built (for any
                            reason). Can be used together with any other build
                            policy. Useful to make sure that any new change
                            introduced in a dependency is incorporated by building
                            again the package. --build=outdated Build from code if
                            the binary is not built with the current recipe or
                            when missing binary package. --build=[pattern] Build
                            always these packages from source, but never build the
                            others. Allows multiple --build parameters. 'pattern'
                            is a fnmatch file pattern of a package reference.
                            Default behavior: If you don't specify anything, it
                            will be similar to '--build=never', but package
                            recipes can override it with their 'build_policy'
                            attribute in the conanfile.py.
```
Hi, are you intending to keep this behavior for Conan 2.0?

From a design point of view, I find it weird, that a recipe can control a build policy, and then that it even has precedence over manually specified values.

I feel that that is a property that should come from the outside, when putting all libraries together. It's like when a CMakeLists.txt file defines its own C++ standard to use, even though this should be an input from the outside, depending on the compiler.

I would be interested to hear where users really need this feature. From my point of view it would be more natural to have it, e.g. in a profile, though I am not sure if `build_policy` is allowed there.


> Hi, are you intending to keep this behavior for Conan 2.0?

We are considering removing these policies for Conan 2.0, or at least redefine their behavior.

The use cases are for example: a driver, that needs to be built from sources always in the target machine. Only the recipe is uploaded, but it is a bit annoying to always fail when users do ``install`` them (maybe even transitively, they are not aware of this driver). Having it in the profile is still on the consumer side, they will still fail, realize they need to build that, add it to their profiles, so a bit inconvenient too. But might be more explicit, and then more aligned with general conan design.

Lets annotate this for Conan 2.0 review.
Hi @memsharded 

Thanks for your quick answer.

I read this part in the docs and still didn't fully understand it. The part

```
--build=never Never build, use binary
packages or fail if a binary package is not found.
```

makes it seem like `--build=never` should fail if _any_ of the packages has to be built, irrespective of their build_policy.

And the part

```
Default behavior: If you don't specify anything, it
will be similar to '--build=never', but package
recipes can override it with their 'build_policy'
attribute in the conanfile.py.
```

makes it seem like the build_policy only affects builds if the `--build` parameter is not set. It sounds like the command line parameter always takes precedence over the conanfile's build_policy.

Maybe we can change the wording of this section in the docs to reflect that behavior a bit better?

> Maybe we can change the wording of this section in the docs to reflect that behavior a bit better?

Yes, certainly this deserves some better wording in the command line help and docs, you are right. This would be an easy contribution, would you like to submit a fix to the command line help yourself, as first step? (docs can be updated later)
> Yes, certainly this deserves some better wording in the command line help and docs, you are right. This would be an easy contribution, would you like to submit a fix to the command line help yourself, as first step? (docs can be updated later)

Yes, I can do that. I have never contributed before though, do I just stick to the rules in the contributing guide? Or is there anything else I should pay attention to?

Yes, this one is very low risk, so those rules should be more than enough. I don't think tests are necessary for this either. Thanks!

Resolving patch:
diff --git a/conans/client/command.py b/conans/client/command.py
--- a/conans/client/command.py
+++ b/conans/client/command.py
@@ -2036,23 +2036,24 @@ def _add_common_install_arguments(parser, build_help, lockfile=True):
                             "Lockfile can be updated if packages change")
 
 
-_help_build_policies = '''Optional, use it to choose if you want to build from sources:
-
-    --build            Build all from sources, do not use binary packages.
-    --build=never      Never build, use binary packages or fail if a binary package is not found.
-    --build=missing    Build from code if a binary package is not found.
-    --build=cascade    Will build from code all the nodes with some dependency being built
-                       (for any reason). Can be used together with any other build policy.
-                       Useful to make sure that any new change introduced in a dependency is
-                       incorporated by building again the package.
-    --build=outdated   Build from code if the binary is not built with the current recipe or
-                       when missing a binary package.
-    --build=[pattern]  Build always these packages from source, but never build the others.
-                       Allows multiple --build parameters. 'pattern' is a fnmatch file pattern
-                       of a package reference.
-
-    Default behavior: If you don't specify anything, it will be similar to '--build={}', but
-    package recipes can override it with their 'build_policy' attribute in the conanfile.py.
+_help_build_policies = '''Optional, specify which packages to build from source. Combining multiple
+    '--build' options on one command line is allowed. For dependencies, the optional 'build_policy'
+    attribute in their conanfile.py takes precedence over the command line parameter.
+    Possible parameters:
+
+    --build            Force build for all packages, do not use binary packages.
+    --build=never      Disallow build for all packages, use binary packages or fail if a binary
+                       package is not found. Cannot be combined with other '--build' options.
+    --build=missing    Build packages from source whose binary package is not found.
+    --build=outdated   Build packages from source whose binary package was not generated from the
+                       latest recipe or is not found.
+    --build=cascade    Build packages from source that have at least one dependency being built from
+                       source.
+    --build=[pattern]  Build packages from source whose package reference matches the pattern. The
+                       pattern uses 'fnmatch' style wildcards.
+
+    Default behavior: If you omit the '--build' option, the 'build_policy' attribute in conanfile.py
+    will be used if it exists, otherwise the behavior is like '--build={}'.
 '''
 
 


Test patch: