Repository: doctest/doctest

Issue #844: When one subcase fails, other independent sibling subcases are not executed anymore
Created: 2024-02-17T14:23:34Z
Description
The documentation states:

For each SUBCASE() the TEST_CASE() is executed from the start

and

Each leaf subcase (a subcase that contains no nested subcases) will be executed exactly once on a separate path of execution from any other leaf subcase (so no leaf subcase can interfere with another).

However, when one leaf subcase fails, other, independent, sibling subcase leafs are not executed anymore. This is highly unsatisfying and not what I expect migrating from Catch2 and based on the documentation above.
Steps to reproduce
Execute the following tests.
#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN
#include "doctest.h"

SCENARIO("My Scenario")
{
    GIVEN("Something")
    {
        WHEN("A")
        {
            THEN("Expected result A-1")
            {
                REQUIRE(0==0);  // Succeeds
            }
            THEN("Expected result A-2")
            {
                REQUIRE(0==1);  // Fails
            }
            THEN("Expected result A-3")
            {
                REQUIRE(0==0);  // Is Not executed
            }
        }
        WHEN("B")
        {
            THEN("Expected result B-1")
            {
                REQUIRE(0==1);  // Is not executed
            }
            THEN("Expected result B-2")
            {
                REQUIRE(0==0);  // Is not executed
            }
        }
    }
}

Test output is:
2 test cases: 1 failed, 1 succeeded

So only 2 test cases are executed instead of 5.
When "A-2" fails, the whole scenario is stopped. Why are "A-3", "B-1" and "B-2" not executed?
They should be totally independent of "A-2".
Expected / desired behaviour
When one leaf fails, still continue with the execution of ancestor and sibling paths to other independent leafs. Compare Catch2 behaviour.
Extra information

doctest version: 2.4.9
Operating System: Ubuntu 20.04.6 LTS x86_64
Compiler+version: Platformio Core 6.1.13 Home 3.4.4 via Platformio IDE 3.3.3 (I assume gcc 9.4.0 for native platform)

Duplicate issue #820: Abort execution of a subcase without halting the entire test case
Created: 2023-10-11T17:41:42Z
Description
Hi! We're using doctest 2.4.9 at Luau for our unit tests. We've run into an issue with subcases that, at this point, is making them not terribly useful for us: if, when executing a subcase, the test case exits without running to completion, all (lexically) subsequent subcases are not run, regardless of nesting depth. This appears most obviously if you use a REQUIRE macro (or otherwise throw) in a test case, but it doesn't require exceptions at all; just a return will do it.
We've made liberal use of REQUIRE macros in order to halt test execution if an assertion fails. Usually this is checking something that will make follow-up CHECK macros fail in a less-obvious manner, like the size of an array before we inspect its first element. Since there doesn't appear to be a way to actually abort execution of a subcase, we don't have many cases where we can use subcases. It would be really helpful if there were some way to abort the execution of a subcase without preventing following subcases from being executed.
Steps to reproduce
This isn't a full example, but this is the kind of test case we'd like to write:
TEST_CASE("test")
{
    // Prints once.
    MESSAGE("test");

    SUBCASE("before_fail")
    {
        // Prints once, as expected.
        MESSAGE("before_fail");
    }

    SUBCASE("fails")
    {
        // Prints once.
        MESSAGE("fails");

        SUBCASE("deeper_fail")
        {
            // Prints once, as expected.
            MESSAGE("deeper_fail");
            REQUIRE(false);
            // After this point, execution halts. No further subcases are invoked.
        }

        SUBCASE("deeper_ok")
        {
            // Should execute, but doesn't.
            MESSAGE("deeper_ok");
        }
    }

    SUBCASE("should_run")
    {
        // Should execute, but doesn't.
        MESSAGE("should_run");
    }
}
If I replace the REQUIRE(false); macro with just return;, I get the same behavior.
Extra information

doctest version: See source here. Based on doctest 2.4.9 with compatibility patches. A diff is included below.
Operating System: macOS 13.5.1
Compiler: Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Platform: arm64

*** doctest-upstream.h	Wed Oct 11 09:56:34 2023
--- doctest-luau.h	Wed Oct 11 09:56:14 2023
***************
*** 3139,3145 ****
--- 3139,3149 ----
  #include <unordered_set>
  #include <exception>
  #include <stdexcept>
+
+ #if !defined(DOCTEST_CONFIG_NO_POSIX_SIGNALS)
  #include <csignal>
+ #endif
+
  #include <cfloat>
  #include <cctype>
  #include <cstdint>
***************
*** 5667,5672 ****
--- 5671,5678 ----
                  std::tm timeInfo;
  #ifdef DOCTEST_PLATFORM_WINDOWS
                  gmtime_s(&timeInfo, &rawtime);
+ #elif defined(DOCTEST_CONFIG_USE_GMTIME_S)
+                 gmtime_s(&rawtime, &timeInfo);
  #else // DOCTEST_PLATFORM_WINDOWS
                  gmtime_r(&rawtime, &timeInfo);
  #endif // DOCTEST_PLATFORM_WINDOWS

Closing comment (2024-08-16T06:23:40Z): Duplicate of #844 (Note that his was actually filed before the other issue, but I started reviewing issues from new to old (switching now so this does not happen again).
Note that this is the intended behavior of doctest.
A simple workaround is to replace REQUIRE with CHECK + if.
I am curious why this feature is so important for you (that is, why this makes doctest "not terribly useful" for you).
If you want to discuss this please do that in the other thread and I will close this issue to make the discussion better.