Repository: conan-io/conan

Issue:
[question] avoid conan create to call tests if it's not necessary
### What is your question?

Hi! I'm setting up the CI/CD for my company to build 3rd party dependencies. I forked the conan-center-index and created a script that essentially calls `conan export`, `conan graph`, `conan create` and `conan upload` given a list of packages and build configugrations.

My issue is that when calling `conan create` on the same package all the tests are called again making the whole operation really long.

Is there way to execute tests only if the packages needs a build?

An example below:

conan export:
```
======== Exporting recipe to the cache ========
zlib/1.2.13: Exporting package recipe: C:\<project>\repo\conan-center-index\recipes\zlib\all\conanfile.py
zlib/1.2.13: exports: File 'conandata.yml' found. Exporting it...
zlib/1.2.13: Calling export_sources()
zlib/1.2.13: Copied 1 '.yml' file: conandata.yml
zlib/1.2.13: Copied 1 '.py' file: conanfile.py
zlib/1.2.13: Copied 1 '.patch' file: 0001-Fix-cmake.patch
zlib/1.2.13: Exported to cache folder: C:\Users\<user>\.conan2\p\zlibdaa2ab943f496\e
zlib/1.2.13: Exported: zlib/1.2.13#df233e6bed99052f285331b9f54d9070 (2024-03-27 16:39:59 UTC)
```

conan graph (to determine early missing dependencies):
```
zlib/1.2.13#df233e6bed99052f285331b9f54d9070:
  ref: zlib/1.2.13#df233e6bed99052f285331b9f54d9070
  id: 1
  recipe: Cache
  package_id: c15c613a1c90445707c636babcad94f4719b0c14
  prev: b2efbbc56d853febf4d58672ce8b26bb
  rrev: df233e6bed99052f285331b9f54d9070
  rrev_timestamp: 1711557659.355807
  prev_timestamp: 1711524555.786842
  remote: None
  binary_remote: None
  build_id: None
  binary: Cache
  invalid_build: False
  info_invalid: None
  name: zlib
  user: None
  channel: None
  url: https://github.com/conan-io/conan-center-index
  license: Zlib
  author: None
  description: A Massively Spiffy Yet Delicately Unobtrusive Compression Library (Also Free, Not to Mention Unencumbered by Patents)
  homepage: https://zlib.net
  build_policy: None
  upload_policy: None
  revision_mode: hash
  provides: None
  deprecated: None
  win_bash: None
  win_bash_run: None
  default_options:
    shared: False
    fPIC: True
  options_description: None
  version: 1.2.13
  topics: ['zlib', 'compression']
  package_type: static-library
  settings:
    os: Android
    os.api_level: 29
    arch: armv8
    compiler: clang
    compiler.version: 9
    build_type: Debug
  options:
    fPIC: True
    shared: False
  options_definitions:
    shared: ['True', 'False']
    fPIC: ['True', 'False']
  generators: []
  python_requires: None
  system_requires:
  recipe_folder: C:\Users\edukaj\.conan2\p\zlibdaa2ab943f496\e
  source_folder: None
  build_folder: None
  generators_folder: None
  package_folder: None
  cpp_info:
    root:
      includedirs: ['include']
      srcdirs: None
      libdirs: ['lib']
      resdirs: None
      bindirs: ['bin']
      builddirs: None
      frameworkdirs: None
      system_libs: None
      frameworks: None
      libs: None
      defines: None
      cflags: None
      cxxflags: None
      sharedlinkflags: None
      exelinkflags: None
      objects: None
      sysroot: None
      requires: None
      properties: None
  conf_info:
  label: zlib/1.2.13
  info:
    settings:
      os: Android
      os.api_level: 29
      arch: armv8
      compiler: clang
      compiler.version: 9
      build_type: Debug
    options:
      fPIC: True
      shared: False
  dependencies:
    2:
      ref: ninja/1.11.1
      run: True
      libs: False
      skip: False
      test: False
      force: False
      direct: True
      build: True
      transitive_headers: None
      transitive_libs: None
      headers: False
      package_id_mode: None
      visible: False
    3:
      ref: cmake/3.28.1
      run: True
      libs: False
      skip: False
      test: False
      force: False
      direct: True
      build: True
      transitive_headers: None
      transitive_libs: None
      headers: False
      package_id_mode: None
      visible: False
    4:
      ref: android-ndk/r21e
      run: True
      libs: False
      skip: False
      test: False
      force: False
      direct: True
      build: True
      transitive_headers: None
      transitive_libs: None
      headers: False
      package_id_mode: None
      visible: False
  context: host
  test: False
```

And when I call conan create, the tests are always executed even if the package doesn't need a build:
```
======== Exporting recipe to the cache ========
zlib/1.2.13: Exporting package recipe: C:\<project>\repo\conan-center-index\recipes\zlib\all\conanfile.py
zlib/1.2.13: exports: File 'conandata.yml' found. Exporting it...
zlib/1.2.13: Calling export_sources()
zlib/1.2.13: Copied 1 '.yml' file: conandata.yml
zlib/1.2.13: Copied 1 '.py' file: conanfile.py
zlib/1.2.13: Copied 1 '.patch' file: 0001-Fix-cmake.patch
zlib/1.2.13: Exported to cache folder: C:\Users\edukaj\.conan2\p\zlibdaa2ab943f496\e
zlib/1.2.13: Exported: zlib/1.2.13#df233e6bed99052f285331b9f54d9070 (2024-03-27 16:46:40 UTC)

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Debug
compiler=msvc
compiler.cppstd=20
compiler.runtime=dynamic
compiler.runtime_type=Debug
compiler.version=193
os=Windows

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=msvc
compiler.cppstd=20
compiler.runtime=dynamic
compiler.runtime_type=Release
compiler.version=193
os=Windows


======== Computing dependency graph ========
Graph root
    cli
Requirements
    zlib/1.2.13#df233e6bed99052f285331b9f54d9070 - Cache

======== Computing necessary packages ========
Requirements
    zlib/1.2.13#df233e6bed99052f285331b9f54d9070:ed44dacde2751b398e4bf9441ff992a70c913c00#6ca1e7aa4648ad6b1d63372a9887837a - Cache

======== Installing packages ========
zlib/1.2.13: Already installed! (1 of 1)
WARN: deprecated: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X:
WARN: deprecated:     'cpp_info.names' used in: zlib/1.2.13

======== Launching test_package ========

======== Computing dependency graph ========
Graph root
    zlib/1.2.13 (test package): C:\<project>\repo\conan-center-index\recipes\zlib\all\test_package\conanfile.py
Requirements
    zlib/1.2.13#df233e6bed99052f285331b9f54d9070 - Cache

======== Computing necessary packages ========
Requirements
    zlib/1.2.13#df233e6bed99052f285331b9f54d9070:ed44dacde2751b398e4bf9441ff992a70c913c00#6ca1e7aa4648ad6b1d63372a9887837a - Cache

======== Installing packages ========
zlib/1.2.13: Already installed! (1 of 1)
WARN: deprecated: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X:
WARN: deprecated:     'cpp_info.names' used in: zlib/1.2.13

======== Testing the package ========
Removing previously existing 'test_package' build folder: C:\<project>\repo\conan-center-index\recipes\zlib\all\test_package\build\msvc-193-x86_64-20-debug
zlib/1.2.13 (test package): Test package build: build\msvc-193-x86_64-20-debug
zlib/1.2.13 (test package): Test package build folder: C:\<project>\repo\conan-center-index\recipes\zlib\all\test_package\build\msvc-193-x86_64-20-debug
zlib/1.2.13 (test package): Writing generators to C:\<project>\repo\conan-center-index\recipes\zlib\all\test_package\build\msvc-193-x86_64-20-debug\generators
zlib/1.2.13 (test package): Generator 'CMakeToolchain' calling 'generate()'
zlib/1.2.13 (test package): CMakeToolchain generated: conan_toolchain.cmake
zlib/1.2.13 (test package): CMakeToolchain generated: CMakePresets.json
zlib/1.2.13 (test package): CMakeToolchain generated: ..\..\..\CMakeUserPresets.json
zlib/1.2.13 (test package): Generator 'CMakeDeps' calling 'generate()'
zlib/1.2.13 (test package): CMakeDeps necessary find_package() and targets for your CMakeLists.txt
    find_package(ZLIB)
    target_link_libraries(... ZLIB::ZLIB)
zlib/1.2.13 (test package): Generator 'VirtualRunEnv' calling 'generate()'
zlib/1.2.13 (test package): Generating aggregated env files
zlib/1.2.13 (test package): Generated aggregated env files: ['conanrun.bat', 'conanbuild.bat']

======== Testing the package: Building ========
zlib/1.2.13 (test package): Calling build()
zlib/1.2.13 (test package): Running CMake.configure()
zlib/1.2.13 (test package): RUN: cmake -G "Visual Studio 17 2022" -DCMAKE_TOOLCHAIN_FILE="C:/<project>/repo/conan-center-index/recipes/zlib/all/test_package/build/msvc-193-x86_64-20-debug/generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="C:/<project>/repo/conan-center-index/recipes/zlib/all/test_package" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" "C:\<project>\repo\conan-center-index\recipes\zlib\all\test_package"
-- Using Conan toolchain: C:/<project>/repo/conan-center-index/recipes/zlib/all/test_package/build/msvc-193-x86_64-20-debug/generators/conan_toolchain.cmake
-- Conan toolchain: CMAKE_GENERATOR_TOOLSET=v143
-- Conan toolchain: C++ Standard 20 with extensions OFF
-- Selecting Windows SDK version 10.0.22000.0 to target Windows 10.0.19045.
-- The C compiler identification is MSVC 19.39.33523.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Professional/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Conan: Target declared 'ZLIB::ZLIB'
-- Configuring done (4.5s)
-- Generating done (0.0s)
-- Build files have been written to: C:/<project>/repo/conan-center-index/recipes/zlib/all/test_package/build/msvc-193-x86_64-20-debug

zlib/1.2.13 (test package): Running CMake.build()
zlib/1.2.13 (test package): RUN: cmake --build "C:\<project>\repo\conan-center-index\recipes\zlib\all\test_package\build\msvc-193-x86_64-20-debug" --config Debug
MSBuild version 17.9.8+b34f75857 for .NET Framework

  Checking Build System
  Building Custom Rule C:/<project>/repo/conan-center-index/recipes/zlib/all/test_package/CMakeLists.txt
  test_package.c
C:\<project>\repo\conan-center-index\recipes\zlib\all\test_package\test_package.c(24,12): warning C4477: 'printf' : format string '%lu' requires an argument of type 'unsigned long', but variadic argument 1 has type 'size_t' [C:\<project>\repo\conan-center-index\r
ecipes\zlib\all\test_package\build\msvc-193-x86_64-20-debug\test_package.vcxproj]
  C:\<project>\repo\conan-center-index\recipes\zlib\all\test_package\test_package.c(24,12):
  consider using '%zu' in the format string

C:\<project>\repo\conan-center-index\recipes\zlib\all\test_package\test_package.c(26,12): warning C4477: 'printf' : format string '%lu' requires an argument of type 'unsigned long', but variadic argument 1 has type 'size_t' [C:\<project>\repo\conan-center-index\r
ecipes\zlib\all\test_package\build\msvc-193-x86_64-20-debug\test_package.vcxproj]
  C:\<project>\repo\conan-center-index\recipes\zlib\all\test_package\test_package.c(26,12):
  consider using '%zu' in the format string

  test_package.vcxproj -> C:\<project>\repo\conan-center-index\recipes\zlib\all\test_package\build\msvc-193-x86_64-20-debug\Debug\test_package.exe
  Building Custom Rule C:/<project>/repo/conan-center-index/recipes/zlib/all/test_package/CMakeLists.txt


======== Testing the package: Executing test ========
zlib/1.2.13 (test package): Running test()
zlib/1.2.13 (test package): RUN: Debug\test_package
Compressed size is: 21
Compressed string is: Conan Package Manager
Compressed size is: 22
Compressed string is: x�s��K�HL�NLOU�M�RE
ZLIB VERSION: 1.2.13
```

### Have you read the CONTRIBUTING guide?

- [X] I've read the CONTRIBUTING guide


Discussion:
Hi @elvisdukaj 

Sure, just call ``conan create . -tf=``  (empty ``--test-folder``) and it will not run the ``test_package``
Hi @memsharded, thank you for the reply!

Doing so, will test be executed in packages that need a build?

i want to be sure that tests are executed only when the package need a build not unconditionally.
> Doing so, will test be executed in packages that need a build?

No, it will disable the test_package.

If you want to do it conditionally, you can:

- Run ``conan create . --build=missing -tf="" --format=json > graph.json``
- Read the graph.json and check if the package was built or not
- If the package was built, run ``conan test`` to execute the ``test_package`` over the created package.

> the tests are always executed even if the package doesn't need a build:

In practice the ``test_package`` will be a very small and fast thing. In Linux systems it should take a couple of seconds, a bit more in Windows, because CMake configure step is slower there. But as this process is typically carried out by CI, it might not be worth it, just to save a few seconds.
This is actually the same question as this one: https://github.com/conan-io/conan/issues/14999

@memsharded Sure, test_packags are small and probably take only a few seconds. But in case of forking conan-center-index this adds up rapidly. Doesn't it make sense to skip the test_package if the package is downloaded? Because I want to exec the tests in the first build in our CI.
@memsharded  thank you very much for the reply and for your valuable hint! Unfortunately, I’ve a situation where I need to compile the same packages for different android variations, in windows, Linux, Darwin, debug and release, c++_shared/c++_static, cppstd 17 and 20, different os_api levels. 

The total time grows up exponentially. I’m just the beginning and the waiting time starts to be not acceptable anymore.

I don’t understand why conan create should run test if the package didn’t change at all.

I’ll try as quick fix to implement what you suggested, and see if I can change the behavior of Conan create.

Do you see any downside in avoiding calling test if the package didn’t build? I cannot think to any.
> I don’t understand why conan create should run test if the package didn’t change at all.

``conan create`` is basically an alias to: ``conan export`` + ``conan install --requires=pkg/1.0 --build=pkg/1.0`` + ``conan test test_package pkg/1.0``

It doesn't really provide much logic rather than combining those commands. The default is always building the package, so the default was to run the test_package, so if running the test_package is too much, this might be split into its different parts or using the ``conan create -tf=""`` 

> Do you see any downside in avoiding calling test if the package didn’t build? I cannot think to any.

Yes, the requested behavior when this was added, many years ago, is that users doing ``create`` they wanted to run a quick check that the package was actually valid and working locally, no matter if built or downloaded from the server (for example, to quickly identify issues in header-only packages that were just uploaded in one platform but fail in others)


@memsharded thanks again for the insight! 


I followed the conan documentation on [how to host my own ConanCenter binaries](https://docs.conan.io/2/devops/conancenter/hosting_binaries.html#creating-and-hosting-your-own-conancenter-binaries)

> * Create a fork of the ConanCenter Github repository: https://github.com/conan-io/conan-center-index
>
> * Create a list of the packages and versions you need for your projects. This list can be added to the fork too, and maintained there (packages can be added and removed with PRs when the teams need them).
>
> * Create a script that first conan export all the packages in your list, then conan create --build=missing them. Do not add user/channel to these packages, it is way simpler to use them as zlib/1.2.13 without user-channel. The user/channel part would be mostly recommended for your own proprietary packages, but not for open source ConanCenter packages.
>
> * Upload your build packages to your own server, that you use in production, instead of ConanCenter.

And this is exactly what I did. But with the new insight about `conan create` I should instead avoid it at all, and replace it with `conan install` and `conan test`, The `export` was called in the previous phase so I can avoid it.

What I should do to speed up the process should be:

* export all the packages;
* for every package (and build variant) call `conan graph info` to early detect missing packages;
* call `conan install --requires=<package>/<version> --build=<package>/<version> --format=json`;
    * if the build is required call `conan test`;
* Upload the package.

I will give it a try and let you know. I still didn't understand how to figure out if the package needs or not a build though.



Patch:
diff --git a/conan/cli/commands/create.py b/conan/cli/commands/create.py
index 36c42372f79..36b96788264 100644
--- a/conan/cli/commands/create.py
+++ b/conan/cli/commands/create.py
@@ -9,6 +9,7 @@
 from conan.cli.printers import print_profiles
 from conan.cli.printers.graph import print_graph_packages, print_graph_basic
 from conan.errors import ConanException
+from conans.client.graph.graph import BINARY_BUILD
 from conans.util.files import mkdir
 
 
@@ -26,11 +27,17 @@ def create(conan_api, parser, *args):
     parser.add_argument("-tf", "--test-folder", action=OnceArgument,
                         help='Alternative test folder name. By default it is "test_package". '
                              'Use "" to skip the test stage')
+    parser.add_argument("-tm", "--test-missing", action='store_true', default=False,
+                        help='Run the test_package checks only if the package is built from source'
+                             ' but not if it already existed (using --build=missing)')
     parser.add_argument("-bt", "--build-test", action="append",
                         help="Same as '--build' but only for the test_package requires. By default"
                              " if not specified it will take the '--build' value if specified")
     args = parser.parse_args(*args)
 
+    if args.test_missing and args.test_folder == "":
+        raise ConanException('--test-folder="" is incompatible with --test-missing')
+
     cwd = os.getcwd()
     path = conan_api.local.get_conanfile_path(args.path, cwd, py=True)
     test_conanfile_path = _get_test_conanfile_path(args.test_folder, path)
@@ -91,6 +98,11 @@ def create(conan_api, parser, *args):
         lockfile = conan_api.lockfile.update_lockfile(lockfile, deps_graph, args.lockfile_packages,
                                                       clean=args.lockfile_clean)
 
+    # If the user provide --test-missing and the binary was not built from source, skip test_package
+    if args.test_missing and deps_graph.root.dependencies\
+            and deps_graph.root.dependencies[0].dst.binary != BINARY_BUILD:
+        test_conanfile_path = None  # disable it
+
     if test_conanfile_path:
         # TODO: We need arguments for:
         #  - decide update policy "--test_package_update"


Test patch:
diff --git a/conans/test/integration/command/create_test.py b/conans/test/integration/command/create_test.py
index 1a8218d2484..9158370df11 100644
--- a/conans/test/integration/command/create_test.py
+++ b/conans/test/integration/command/create_test.py
@@ -811,3 +811,53 @@ def test(self):
     c.run("create .")
     # Ensure that creating a deps graph does not break the testing
     assert "pyreq/1.0 (test package): 42!!!" in c.out
+
+
+def test_create_test_package_only_build():
+    c = TestClient()
+    c.save({"conanfile.py": GenConanfile("pkg", "0.1"),
+            "test_package/conanfile.py": GenConanfile().with_test("self.output.info('TEST1!!!')"),
+            "test_package2/conanfile.py": GenConanfile().with_test("self.output.info('TEST2!!!')")})
+    # As it doesn't exist, it builds and test it
+    c.run("create . -tm")
+    assert "Testing the package" in c.out
+    assert "TEST1!!!" in c.out
+    # this will not create the binary, so it won't test it
+    c.run("create . --build=missing --test-missing")
+    assert "Testing the package" not in c.out
+    assert "TEST" not in c.out
+    c.run("create . -tf=test_package2 -tm")
+    assert "Testing the package" in c.out
+    assert "TEST2!!!" in c.out
+    assert "TEST1!!!" not in c.out
+    c.run("create . -tf=test_package2 --build=missing --test-missing")
+    assert "Testing the package" not in c.out
+    assert "TEST2!!!" not in c.out
+    assert "TEST1!!!" not in c.out
+
+    # error
+    c.run("create . -tm -tf=", assert_error=True)
+    assert '--test-folder="" is incompatible with --test-missing' in c.out
+
+
+def test_create_test_package_only_build_python_require():
+    c = TestClient()
+    test = textwrap.dedent("""
+        from conan import ConanFile
+
+        class Tool(ConanFile):
+            python_requires = "tested_reference_str"
+            def test(self):
+                self.output.info("TEST!!!!")
+        """)
+    c.save({"conanfile.py": GenConanfile("pkg", "0.1").with_package_type("python-require"),
+            "test_package/conanfile.py": test})
+    c.run("create .")
+    assert "Testing the package" in c.out
+    assert "pkg/0.1 (test package): TEST!!!" in c.out
+    c.run("create . -tm")
+    assert "Testing the package" in c.out
+    assert "pkg/0.1 (test package): TEST!!!" in c.out
+    c.run("create . -tm --build=missing")
+    assert "Testing the package" in c.out
+    assert "pkg/0.1 (test package): TEST!!!" in c.out