Repository: conan-io/conan

Issue:
CMake helper to get CTest to output on failure
I'd like to be able to configure my `conanfile.py` so that I can invoke `ctest --output-on-failure` in some fashion. After chatting with on cpplang.slack.com, it was noted that it doesn't look like this is currently possible, and so I'll need to invoke this manually.

Is such a feature on the roadmap, and if not, could it please be added in a future release?


Discussion:
I don't know much about this ``ctest --output-on-failure``, but it might be possible, as some arguments to ``cmake.test()`` method, or maybe as another ``cmake`` method. What is the current workaround, can you provide a full working ``build()`` method with a custom ``self.run("ctest ...`` or whatever to give an idea of the feature? Thanks!
Right now, my workaround is to run this on the command line after `conan
build .. --build`:

```bash
ctest -j `nproc` --output-on-failure
```

On Sat., 8 Sep. 2018, 11:56 James, <notifications@github.com> wrote:

> I don't know much about this ctest --output-on-failure, but it might be
> possible, as some arguments to cmake.test() method, or maybe as another
> cmake method. What is the current workaround, can you provide a full
> working build() method with a custom self.run("ctest ... or whatever to
> give an idea of the feature? Thanks!
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/conan-io/conan/issues/3511#issuecomment-419632998>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AIOSnFM-NqlMNHfUpd3NkkstevLbBo48ks5uY6JqgaJpZM4Wfzvk>
> .
>

Hi @cjdb 

Why not something like this?

```python
def build(self):
    cmake = CMake(self):
    cmake.configure(...)
    cmake.build(...)
    if self.should_test:
        self.run("ctest -j %s --output-on-failure" % tools.cpu_count())
```

Then you can run it with:
```bash
$ conan build .. --build
$ conan build .. --test
```
Current implementation of ``cmake.test()`` is ``cmake --build . --target RUN_TESTS`` (or ``--target test`` depends on the build system), so it seems too different to be re-purposed for calling ``ctest``.

What do you think?
I like it!

I understand that Conan can invoke parallel builds with build systems such
as `make` (which are not parallel by default).

Does this happen for testing too? `nproc` is a Unix tool for getting the
number of processors, and I think that will inhibit usage on Windows
systems.

On Mon., 10 Sep. 2018, 09:50 James, <notifications@github.com> wrote:

> Hi @cjdb <https://github.com/cjdb>
>
> Why not something like this?
>
> def build(self):
>     cmake = CMake(self):
>     cmake.configure(...)
>     cmake.build(...)
>     if self.should_test:
>         self.run("ctest -j `nproc` --output-on-failure")
>
> Then you can run it with:
>
> $ conan build .. --build
> $ conan build .. --test
>
> Current implementation of cmake.test() is cmake --build . --target
> RUN_TESTS (or --target test depends on the build system), so it seems too
> different to be re-purposed for calling ctest.
>
> What do you think?
>
> —
> You are receiving this because you were mentioned.
>
>
> Reply to this email directly, view it on GitHub
> <https://github.com/conan-io/conan/issues/3511#issuecomment-419837350>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AIOSnK77kVlMxlUGDZB83eIPw_wrXUUbks5uZifkgaJpZM4Wfzvk>
> .
>

> I understand that Conan can invoke parallel builds with build systems such
as `make` (which are not parallel by default).

Yes, the built-in build helpers, like MSBuild(), CMake(), AutoToolsBuildEnvironment already implement calling the build systems in parallel (check ``parallel=True`` argument default value in their constructor), and will map to the right -j, /MP build system arguments. It will be using the ``tools.cpu_count()`` function, which will detect the number of processor in a portable way, but is also configurable in ``conan.conf``, so the user can decide if they want all their cores to build. 

I would recommend using it also in your scripts, as ``nproc`` might be a bit aggressive for users that want to leave some cores free to keep working while installing and building conan packages. Besides, it is not portable, as you pointed out, so it won't work in Windows.
from the [doc](https://cmake.org/cmake/help/v3.13/manual/ctest.1.html):
```
--output-on-failure
Output anything outputted by the test program if the test should fail. This option can also be enabled by setting the CTEST_OUTPUT_ON_FAILURE environment variable
```
it means we can just set `CTEST_OUTPUT_ON_FAILURE` environment variable

Resolving patch:
diff --git a/conans/client/build/cmake.py b/conans/client/build/cmake.py
--- a/conans/client/build/cmake.py
+++ b/conans/client/build/cmake.py
@@ -233,12 +233,17 @@ def install(self, args=None, build_dir=None):
                                  "defined" % cmake_install_prefix_var_name)
         self._build(args=args, build_dir=build_dir, target="install")
 
-    def test(self, args=None, build_dir=None, target=None):
+    def test(self, args=None, build_dir=None, target=None, output_on_failure=False):
         if not self._conanfile.should_test:
             return
         if not target:
             target = "RUN_TESTS" if self.is_multi_configuration else "test"
-        self._build(args=args, build_dir=build_dir, target=target)
+
+        env = {'CTEST_OUTPUT_ON_FAILURE': '1' if output_on_failure else '0'}
+        if self.parallel:
+            env['CTEST_PARALLEL_LEVEL'] = str(cpu_count(self._conanfile.output))
+        with tools.environment_append(env):
+            self._build(args=args, build_dir=build_dir, target=target)
 
     @property
     def verbose(self):


Test patch:
diff --git a/conans/test/unittests/client/build/cmake_test.py b/conans/test/unittests/client/build/cmake_test.py
--- a/conans/test/unittests/client/build/cmake_test.py
+++ b/conans/test/unittests/client/build/cmake_test.py
@@ -1160,3 +1160,22 @@ def test_vcvars_applied(self, generator, compiler, version):
             vcvars_mock.__exit__ = mock.MagicMock(return_value=None)
             cmake.build()
             self.assertTrue(vcvars_mock.called, "vcvars weren't called")
+
+    def test_ctest_variables(self):
+        conanfile = ConanFileMock()
+        settings = Settings.loads(default_settings_yml)
+        settings.os = "Windows"
+        settings.compiler = "Visual Studio"
+        settings.compiler.version = "14"
+        conanfile.settings = settings
+
+        cmake = CMake(conanfile, parallel=False, generator="NMake Makefiles")
+        cmake.test()
+        self.assertEquals(conanfile.captured_env["CTEST_OUTPUT_ON_FAILURE"], "0")
+        self.assertNotIn("CTEST_PARALLEL_LEVEL", conanfile.captured_env)
+
+        with tools.environment_append({"CONAN_CPU_COUNT": "666"}):
+            cmake = CMake(conanfile, parallel=True, generator="NMake Makefiles")
+            cmake.test(output_on_failure=True)
+            self.assertEquals(conanfile.captured_env["CTEST_OUTPUT_ON_FAILURE"], "1")
+            self.assertEquals(conanfile.captured_env["CTEST_PARALLEL_LEVEL"], "666")