Repository: Behat/Behat (php)

Issue:
Support PHP config for "tester" options
We haven't yet exposed an explicit PHP interface for configuring the `strict` or `stop-on-failure` runtime behaviour. I started looking at this and realised there's a couple of others:

```yaml
default:
  testers:
    # "Sets the strict mode for result interpretation" (which basically means fail on pending or undefined)
    strict: true
    stop_on_failure: true
    # "Tells tester to skip all tests" - tbh I'm not certain why you'd want this, and it doesn't appear to have feature coverage. 
    # Maybe for a profile that always runs equivalent to `--dry-run`?
    skip: true
  calls:
     # This is in a separate section of the configuration, but feels like it might be logically related to configuring
     # how the testers behave e.g. whether they pass or fail depending on particular conditions.
     # Note this is the only supported config key under `calls` at the moment, we might in future want to
     # add separate settings for collecting / handling deprecations which need to be configured alongside
     # error_reporting but don't really feel like they belong under something called `calls`.
    error_reporting: 32757
```

Arguably these are all behaviours of Behat for a specific `Profile` so they could be direct setters on the `Profile` config object. However that class is already getting quite heavy, and we probably don't want to add a lot more methods to that.

I think we should probably group these together into a new config object, either using `withXXX(true|false)` for each option:

```php
<?php
use Behat\Config\Config;
use Behat\Config\Profile;
use Behat\Config\TesterOptions;

return (new Config())
    ->withProfile((new Profile('default'))
        ->withTesterOptions((new TesterOptions)
            ->withStrictTesters(true)
            ->withStopOnFailure(false)
            ->withSkipAllTests(true)
            ->withErrorReporting(32757)
        )
    );
```

Or possibly with named methods for the on/off boolean states - I find these a little clearer to read, but they are obviously then harder if you wanted to set them based on some variable:

```php
<?php
use Behat\Config\Config;
use Behat\Config\Profile;
use Behat\Config\TesterOptions;

return (new Config())
    ->withProfile((new Profile('default'))
        ->withTesterOptions((new TesterOptions)
            ->failOnPendingOrUndefined() # or passOnPendingOrUndefined()
            ->stopOnFailure() # or continueOnFailure()
            ->skipAllTests() # I'm not convinced this one needs a negation
            ->withErrorReporting(32757)
        )
    );
```

I'm not sure how best to name this group of settings. 

`->withTesterOptions(new TesterOptions` roughly follows the underlying `testers` config key and the `Behat\Testwork\Tester\ServiceContainer\TesterExtension` that uses them. But I'm not sure if it's obvious to an outside user - `testers` on it's own might suggest the concept that `contexts` actually covers, but `TesterOptions` doesn't fit the pattern of the rest of our config.

These options are really to do with how Behat behaves when running tests. Apart from `skip`, they are sort of `ResultHandling` - `error_reporting` obviously runs code before the step executes and changes how the step behaves, but ultimately controls whether e.g. a PHP warning during a step causes the test to fail or not. So maybe `->withResultHandling((new ResultHandling)->stopOnFailure()` etc?

@carlos-granados WDYT?

Pull request description:
Support (and convert) PHP config for tester options
This branch is based on #1625 so that should be reviewed and merged first.

Adds support for configuring the tester and error_reporting options through PHP, and for converting those configs from YAML.

This includes automatically removing default arguments when generating PHP config methods from YAML so that you get e.g. `->withStopOnFailure()` rather than `->withStopOnFailure(true)`. That change also applies to a couple of existing methods - as shown in the updated convert_config feature file - and as a byproduct means that the config conversion process is now stricter internally about making sure that called methods exist and have been provided with the correct argument list.


Fixes #1624

Patch:
diff --git a/features/convert_config.feature b/features/convert_config.feature
index 634c1b46..069e98fa 100644
--- a/features/convert_config.feature
+++ b/features/convert_config.feature
@@ -366,7 +366,7 @@ Feature: Convert config
 
       return (new Config())
           ->withProfile((new Profile('default'))
-              ->withPrintUnusedDefinitions(true));
+              ->withPrintUnusedDefinitions());
       """
     And the temp "unused_definitions.yaml" file should have been removed
 
@@ -434,6 +434,36 @@ Feature: Convert config
       """
     And the temp "path_options.yaml" file should have been removed
 
+  Scenario: Tester options
+    Given I copy the "tester_options.yaml" file to the temp folder
+    When I run behat with the following additional options:
+      | option   | value                                    |
+      | --config | {SYSTEM_TMP_DIR}/tester_options.yaml |
+    Then the temp "tester_options.php" file should be like:
+      """
+      <?php
+
+      use Behat\Config\Config;
+      use Behat\Config\Profile;
+      use Behat\Config\TesterOptions;
+
+      return (new Config())
+          ->withProfile(new Profile('default'))
+          ->withProfile((new Profile('ignore-errors'))
+              ->withTesterOptions((new TesterOptions())
+                  ->withErrorReporting(E_ALL & ~E_DEPRECATED)))
+          ->withProfile((new Profile('not-strict'))
+              ->withTesterOptions((new TesterOptions())
+                  ->withStrictResultInterpretation(false)))
+          ->withProfile((new Profile('complete'))
+              ->withTesterOptions((new TesterOptions())
+                  ->withStrictResultInterpretation()
+                  ->withStopOnFailure(false)
+                  ->withSkipAllTests()
+                  ->withErrorReporting(E_ALL & ~(E_WARNING | E_NOTICE | E_DEPRECATED))));
+      """
+    And the temp "tester_options.yaml" file should have been removed
+
   Scenario: Full configuration
     When I copy the "full_configuration.yaml" file to the temp folder
     And I copy the "imported.yaml" file to the temp folder
@@ -455,6 +485,7 @@ Feature: Convert config
       use Behat\Config\Formatter\ProgressFormatter;
       use Behat\Config\Profile;
       use Behat\Config\Suite;
+      use Behat\Config\TesterOptions;
       use Behat\Testwork\Output\Printer\Factory\OutputFactory;
 
       return (new Config())
@@ -470,8 +501,10 @@ Feature: Convert config
                   ->withOutputVerbosity(OutputFactory::VERBOSITY_VERBOSE))
               ->withFilter(new NameFilter('john'))
               ->withFilter(new RoleFilter('admin'))
-              ->withPrintUnusedDefinitions(true)
+              ->withPrintUnusedDefinitions()
               ->withPathOptions(printAbsolutePaths: true)
+              ->withTesterOptions((new TesterOptions())
+                  ->withStrictResultInterpretation())
               ->withExtension(new Extension('custom_extension.php'))
               ->withSuite((new Suite('my_suite'))
                   ->addContext(
@@ -483,7 +516,9 @@ Feature: Convert config
                   ->withPaths('one.feature')
                   ->withFilter(new TagFilter('@run'))))
           ->withProfile((new Profile('other'))
-              ->disableFormatter('pretty'))
+              ->disableFormatter('pretty')
+              ->withTesterOptions((new TesterOptions())
+                  ->withErrorReporting(E_ERROR)))
           ->withPreferredProfile('other');
       """
     And the temp "imported.php" file should be like:
diff --git a/src/Behat/Config/Config.php b/src/Behat/Config/Config.php
index 0996b24d..515fa4a0 100644
--- a/src/Behat/Config/Config.php
+++ b/src/Behat/Config/Config.php
@@ -71,6 +71,7 @@ final class Config implements ConfigInterface, ConfigConverterInterface
         foreach ($this->settings as $settingsName => $settings) {
             if ($settingsName === self::PREFERRED_PROFILE_NAME_SETTING) {
                 $expr = ConfigConverterTools::addMethodCall(
+                    self::class,
                     self::PREFERRED_PROFILE_FUNCTION,
                     [$settings],
                     $expr
@@ -82,6 +83,7 @@ final class Config implements ConfigInterface, ConfigConverterInterface
                     $arguments = [$settings];
                 }
                 $expr = ConfigConverterTools::addMethodCall(
+                    self::class,
                     self::IMPORT_FUNCTION,
                     $arguments,
                     $expr
@@ -89,6 +91,7 @@ final class Config implements ConfigInterface, ConfigConverterInterface
             } else {
                 $profile = new Profile($settingsName, $settings ?? []);
                 $expr = ConfigConverterTools::addMethodCall(
+                    self::class,
                     self::PROFILE_FUNCTION,
                     [$profile->toPhpExpr()],
                     $expr
diff --git a/src/Behat/Config/Converter/ConfigConverterTools.php b/src/Behat/Config/Converter/ConfigConverterTools.php
index 7caf07c5..e3afdc4b 100644
--- a/src/Behat/Config/Converter/ConfigConverterTools.php
+++ b/src/Behat/Config/Converter/ConfigConverterTools.php
@@ -4,7 +4,6 @@ declare(strict_types=1);
 
 namespace Behat\Config\Converter;
 
-use Behat\Testwork\Output\Printer\Factory\OutputFactory;
 use PhpParser\BuilderFactory;
 use PhpParser\Node\Expr;
 use PhpParser\Node\Expr\New_;
@@ -12,18 +11,74 @@ use PhpParser\Node\Name\FullyQualified;
 use PhpParser\Node\Stmt\Use_;
 use ReflectionClass;
 use ReflectionClassConstant;
+use ReflectionMethod;
 
 class ConfigConverterTools
 {
     private static ?BuilderFactory $builderFactory = null;
 
-    public static function addMethodCall(string $methodName, array $arguments, Expr $expr): Expr
+    public static function addMethodCall(string $class, string $methodName, array $arguments, Expr $expr): Expr
     {
         $builderFactory = self::getBuilderFactory();
+        $arguments = self::removeDefaultArguments((new ReflectionClass($class))->getMethod($methodName), $arguments);
         $args = $builderFactory->args(self::replaceClassReferences($arguments));
         return $builderFactory->methodCall($expr, $methodName, $args);
     }
 
+    private static function removeDefaultArguments(ReflectionMethod $method, array $arguments): array
+    {
+        $hasRemovedAny = false;
+        $result = [];
+        foreach ($method->getParameters() as $index => $parameter) {
+            if ($parameter->isVariadic()) {
+                // We can't do anything useful (easily) for methods with variadics, just return the originals
+                return $arguments;
+            }
+
+            // Config objects can export their method arguments as positional or named arguments.
+            $argKey = array_key_exists($parameter->getName(), $arguments) ? $parameter->getName() : $index;
+
+            // The Config object must provide a value for every argument (defaults are stripped here). If we are
+            // missing arguments, that implies the ->toPhpExpr implementation is out of sync with the method definition.
+            assert(
+                array_key_exists($argKey, $arguments),
+                sprintf(
+                    'Missing argument #%d (%s) for %s::%s',
+                    $index,
+                    $parameter->getName(),
+                    $method->getDeclaringClass()->getName(),
+                    $method->getName(),
+                ),
+            );
+
+            $argValue = $arguments[$argKey];
+            if ($parameter->isDefaultValueAvailable() && $parameter->getDefaultValue() === $argValue) {
+                // We can skip this, it has the default value. That means we'll need to specify any following arguments
+                // by name even if we were given them as positional.
+                $hasRemovedAny = true;
+            } else {
+                $result[($hasRemovedAny ? $parameter->getName() : $argKey)] = $argValue;
+            }
+
+            // Record that this argument has been used
+            unset($arguments[$argKey]);
+        }
+
+        // We should have applied all arguments that were provided. If not, that implies the ->toPhpExpr implementation
+        // is out of sync with the method definition.
+        assert(
+            $arguments === [],
+            sprintf(
+                'Too many arguments provided for %s::%s (%s)',
+                $method->getDeclaringClass()->getName(),
+                $method->getName(),
+                implode(', ', array_keys($arguments)),
+            ),
+        );
+
+        return $result;
+    }
+
     public static function createObject(string $className): New_
     {
         $builderFactory = self::getBuilderFactory();
@@ -77,4 +132,79 @@ class ConfigConverterTools
         // Can't find a constant for it, just return the original value
         return $settingValue;
     }
+
+    public static function errorReportingToConstants(mixed $value): Expr
+    {
+        $builderFactory = self::getBuilderFactory();
+
+        if (! is_int($value)) {
+            return $builderFactory->val($value);
+        }
+
+        // First remove the old E_STRICT value from both E_ALL and from the actual configured value.
+        // This will make the behaviour consistent across all supported PHP versions. Since 8.4.0,
+        // use of E_STRICT is deprecated and the value is no longer included in E_ALL. However,
+        // the bit may still be set in the error_reporting integer in a user's YAML file.
+        // Since 8.0, PHP cannot actually ever trigger an E_STRICT error, so the setting of the bit
+        // is actually irrelevant and we can safely ignore it.
+        $value = $value & ~2048;
+
+        if ($value === (E_ALL & ~2048)) {
+            // The value is equivalent to E_ALL (with or without the E_STRICT bit set, it doesn't matter).
+            return $builderFactory->constFetch('E_ALL');
+        }
+
+        if ($value === 0) {
+            // error_reporting is disabled
+            return $builderFactory->val(0);
+        }
+
+        // Split the value into constant expressions for the bit flags that are on and off.
+        $flagsOn = $flagsOff = [];
+        $remainingValue = $value;
+        foreach (self::listErrorLevelConstants() as $constName => $constValue) {
+            $remainingValue = $remainingValue & ~$constValue;
+
+            if ($value & $constValue) {
+                $flagsOn[] = $builderFactory->constFetch($constName);
+            } else {
+                $flagsOff[] = $builderFactory->constFetch($constName);
+            }
+        }
+
+        if ($remainingValue !== 0) {
+            // This is an edge case, their value includes a number that is not defined as a PHP error_reporting level
+            // Keep it as a constant value so that the configuration doesn't change and the user can review it
+            $flagsOn[] = $builderFactory->val($remainingValue);
+        }
+
+        if (count($flagsOff) < count($flagsOn)) {
+            // Most flags are on, it will be more readable and future-proof to start from E_ALL and negate
+            return new Expr\BinaryOp\BitwiseAnd(
+                $builderFactory->constFetch('E_ALL'),
+                new Expr\BitwiseNot(self::buildBitwiseFlagExpression(...$flagsOff)),
+            );
+        }
+
+        // Most flags are off, so just specify the ones that are on
+        return self::buildBitwiseFlagExpression(...$flagsOn);
+    }
+
+    private static function listErrorLevelConstants(): iterable
+    {
+        foreach (get_defined_constants(true)['Core'] as $constName => $constValue) {
+            if (str_starts_with($constName, 'E_') && $constName !== 'E_ALL' && $constName !== 'E_STRICT') {
+                yield $constName => $constValue;
+            }
+        }
+    }
+
+    private static function buildBitwiseFlagExpression(Expr ...$expressions): Expr
+    {
+        $expr = array_shift($expressions);
+        while ($expressions !== []) {
+            $expr = new Expr\BinaryOp\BitwiseOr($expr, array_shift($expressions));
+        }
+        return $expr;
+    }
 }
diff --git a/src/Behat/Config/Formatter/Formatter.php b/src/Behat/Config/Formatter/Formatter.php
index 027dee6a..a73e9a74 100644
--- a/src/Behat/Config/Formatter/Formatter.php
+++ b/src/Behat/Config/Formatter/Formatter.php
@@ -139,6 +139,7 @@ class Formatter implements FormatterConfigInterface, ConfigConverterInterface
                     $setting = ConfigConverterTools::findReferenceToClassConstant(OutputFactory::class, $setting);
                 }
                 $expr = ConfigConverterTools::addMethodCall(
+                    self::class,
                     $functionName,
                     [$setting],
                     $expr
diff --git a/src/Behat/Config/Profile.php b/src/Behat/Config/Profile.php
index 0cee6133..e5f5351d 100644
--- a/src/Behat/Config/Profile.php
+++ b/src/Behat/Config/Profile.php
@@ -40,6 +40,7 @@ final class Profile implements ConfigConverterInterface
     private const SUITE_FUNCTION = 'withSuite';
     private const PATH_OPTIONS_FUNCTION = 'withPathOptions';
     private const PRINT_ABSOLUTE_PATHS_PARAMETER = 'printAbsolutePaths';
+    private const TESTER_OPTIONS_FUNCTION = 'withTesterOptions';
 
     public function __construct(
         private string $name,
@@ -113,6 +114,16 @@ final class Profile implements ConfigConverterInterface
         return $this;
     }
 
+    public function withTesterOptions(TesterOptions $options): self
+    {
+        // Tester options are split between multiple settings keys due to implementation details in Behat
+        foreach ($options->toArray() as $group => $groupSettings) {
+            $this->settings[$group] = $groupSettings;
+        }
+
+        return $this;
+    }
+
     public function toArray(): array
     {
         return $this->settings;
@@ -130,6 +141,7 @@ final class Profile implements ConfigConverterInterface
         $this->addFiltersToExpr($expr);
         $this->addUnusedDefinitionsToExpr($expr);
         $this->addPathOptionsToExpr($expr);
+        $this->addTesterOptionsToExpr($expr);
         $this->addExtensionsToExpr($expr);
         $this->addSuitesToExpr($expr);
 
@@ -151,6 +163,7 @@ final class Profile implements ConfigConverterInterface
         foreach ($this->settings[self::FORMATTERS_SETTING] as $name => $formatterSettings) {
             if ($formatterSettings === false) {
                 $expr = ConfigConverterTools::addMethodCall(
+                    self::class,
                     self::DISABLE_FORMATTER_FUNCTION,
                     [$name],
                     $expr
@@ -172,6 +185,7 @@ final class Profile implements ConfigConverterInterface
                     default => $formatterSettings === true ? new Formatter($name) : new Formatter($name, $formatterSettings),
                 };
                 $expr = ConfigConverterTools::addMethodCall(
+                    self::class,
                     self::FORMATTER_FUNCTION,
                     [$formatter->toPhpExpr()],
                     $expr
@@ -196,6 +210,7 @@ final class Profile implements ConfigConverterInterface
             };
             if ($filter !== null) {
                 $expr = ConfigConverterTools::addMethodCall(
+                    self::class,
                     self::FILTER_FUNCTION,
                     [$filter->toPhpExpr()],
                     $expr
@@ -217,6 +232,7 @@ final class Profile implements ConfigConverterInterface
             return;
         }
         $expr = ConfigConverterTools::addMethodCall(
+            self::class,
             self::UNUSED_DEFINITIONS_FUNCTION,
             [$this->settings[self::DEFINITIONS_SETTING][self::PRINT_UNUSED_DEFINITIONS_SETTING]],
             $expr
@@ -233,6 +249,7 @@ final class Profile implements ConfigConverterInterface
             return;
         }
         $expr = ConfigConverterTools::addMethodCall(
+            self::class,
             self::PATH_OPTIONS_FUNCTION,
             [self::PRINT_ABSOLUTE_PATHS_PARAMETER => $this->settings[self::PATH_OPTIONS_SETTING][self::PRINT_ABSOLUTE_PATHS_SETTING]],
             $expr
@@ -243,6 +260,21 @@ final class Profile implements ConfigConverterInterface
         }
     }
 
+    private function addTesterOptionsToExpr(Expr &$expr): void
+    {
+        $optionsObject = TesterOptions::consumeSettingsFromProfile($this->settings);
+        if ($optionsObject === null) {
+            return;
+        }
+
+        $expr = ConfigConverterTools::addMethodCall(
+            self::class,
+            self::TESTER_OPTIONS_FUNCTION,
+            [$optionsObject->toPhpExpr()],
+            $expr,
+        );
+    }
+
     private function addExtensionsToExpr(Expr &$expr): void
     {
         if (!isset($this->settings[self::EXTENSIONS_SETTING])) {
@@ -252,6 +284,7 @@ final class Profile implements ConfigConverterInterface
             $extensionObject = new Extension($name, $extensionSettings ?? []);
 
             $expr = ConfigConverterTools::addMethodCall(
+                self::class,
                 self::EXTENSION_FUNCTION,
                 [$extensionObject->toPhpExpr()],
                 $expr
@@ -268,6 +301,7 @@ final class Profile implements ConfigConverterInterface
         foreach ($this->settings[self::SUITES_SETTING] as $name => $suiteSettings) {
             $suiteObject = new Suite($name, $suiteSettings ?? []);
             $expr = ConfigConverterTools::addMethodCall(
+                self::class,
                 self::SUITE_FUNCTION,
                 [$suiteObject->toPhpExpr()],
                 $expr
diff --git a/src/Behat/Config/Suite.php b/src/Behat/Config/Suite.php
index c97f47bb..4c0b4e72 100644
--- a/src/Behat/Config/Suite.php
+++ b/src/Behat/Config/Suite.php
@@ -117,7 +117,7 @@ final class Suite implements ConfigConverterInterface
 
         if (!$hasAnyWithArguments) {
             // All the contexts are just class names, we can add them as a single `->withContexts` call
-            $expr = ConfigConverterTools::addMethodCall(self::WITH_CONTEXTS_FUNCTION, $contexts, $expr);
+            $expr = ConfigConverterTools::addMethodCall(self::class, self::WITH_CONTEXTS_FUNCTION, $contexts, $expr);
             return;
         }
 
@@ -131,9 +131,9 @@ final class Suite implements ConfigConverterInterface
                     array_shift($contextConfig),
                 ];
             } else {
-                $args = [$contextConfig];
+                $args = [$contextConfig, []];
             }
-            $expr = ConfigConverterTools::addMethodCall(self::ADD_CONTEXT_FUNCTION, $args, $expr);
+            $expr = ConfigConverterTools::addMethodCall(self::class, self::ADD_CONTEXT_FUNCTION, $args, $expr);
         }
     }
 
@@ -141,6 +141,7 @@ final class Suite implements ConfigConverterInterface
     {
         if (isset($this->settings[self::PATHS_SETTING])) {
             $expr = ConfigConverterTools::addMethodCall(
+                self::class,
                 self::PATHS_FUNCTION,
                 $this->settings[self::PATHS_SETTING],
                 $expr
@@ -162,6 +163,7 @@ final class Suite implements ConfigConverterInterface
                 };
                 if ($filter !== null) {
                     $expr = ConfigConverterTools::addMethodCall(
+                        self::class,
                         self::FILTER_FUNCTION,
                         [$filter->toPhpExpr()],
                         $expr


Test patch:
diff --git a/src/Behat/Config/TesterOptions.php b/src/Behat/Config/TesterOptions.php
new file mode 100644
index 00000000..8f1622a7
--- /dev/null
+++ b/src/Behat/Config/TesterOptions.php
@@ -0,0 +1,148 @@
+<?php
+
+namespace Behat\Config;
+
+use Behat\Config\Converter\ConfigConverterTools;
+use PhpParser\Node\Expr;
+
+final class TesterOptions implements ConfigConverterInterface
+{
+    private const TESTERS_SETTINGS_GROUP = 'testers';
+
+    private const CALLS_SETTINGS_GROUP = 'calls';
+
+    private const STRICT_SETTING = 'strict';
+
+    private const STOP_ON_FAILURE_SETTING = 'stop_on_failure';
+
+    private const SKIP_SETTING = 'skip';
+
+    private const ERROR_REPORTING_SETTING = 'error_reporting';
+
+    private const FUNCTION_NAMES_PER_SETTING = [
+        self::CALLS_SETTINGS_GROUP => [
+            self::ERROR_REPORTING_SETTING => 'withErrorReporting',
+        ],
+        self::TESTERS_SETTINGS_GROUP => [
+            self::STOP_ON_FAILURE_SETTING => 'withStopOnFailure',
+            self::SKIP_SETTING => 'withSkipAllTests',
+            self::STRICT_SETTING => 'withStrictResultInterpretation',
+        ],
+    ];
+
+    /**
+     * Used when converting config to PHP, to move relevant Profile settings into this object
+     *
+     * @internal
+     */
+    public static function consumeSettingsFromProfile(&$profileSettings): ?self
+    {
+        // These settings live under multiple array keys. We don't want to expose/couple these details to the Profile,
+        // but we need the Profile to be able to detect whether there *are* any TesterOptions to convert.
+
+        // Build an array of relevant settings groups. Remove any that are found from the settings array in the Profile
+        // object, so that the config converter recognises they have been handled.
+        $settings = [];
+        foreach ([self::TESTERS_SETTINGS_GROUP, self::CALLS_SETTINGS_GROUP] as $group) {
+            if (isset($profileSettings[$group])) {
+                $settings[$group] = $profileSettings[$group];
+                unset($profileSettings[$group]);
+            }
+        }
+
+        if ($settings === []) {
+            // There are no relevant settings, so the config does not need to call ->withTesterOptions on the Profile
+            return null;
+        }
+
+        return new self($settings);
+    }
+
+    public function __construct(
+        private array $settings = [],
+    ) {
+
+    }
+
+    public function toArray(): array
+    {
+        return $this->settings;
+    }
+
+    /**
+     * Behat will convert PHP warnings / errors during steps to exceptions if they match this error_reporting level
+     */
+    public function withErrorReporting(int $errorReporting): self
+    {
+        $this->settings[self::CALLS_SETTINGS_GROUP][self::ERROR_REPORTING_SETTING] = $errorReporting;
+
+        return $this;
+    }
+
+
+    /**
+     * Control whether Behat should fail on undefined or pending steps (equivalent to the `--strict` CLI flag)
+     */
+    public function withStrictResultInterpretation(bool $strict = true): self
+    {
+        $this->settings[self::TESTERS_SETTINGS_GROUP][self::STRICT_SETTING] = $strict;
+
+        return $this;
+    }
+
+    /**
+     * Control whether Behat should actually execute steps (equivalent to the `--dry-run` CLI flag)
+     */
+    public function withSkipAllTests(bool $skip = true): self
+    {
+        $this->settings[self::TESTERS_SETTINGS_GROUP][self::SKIP_SETTING] = $skip;
+
+        return $this;
+    }
+
+    /**
+     * Control whether Behat should stop after the first failing scenario (equivalent to `--stop-on-failure` on CLI)
+     */
+    public function withStopOnFailure(bool $stopOnFailure = true): self
+    {
+        $this->settings[self::TESTERS_SETTINGS_GROUP][self::STOP_ON_FAILURE_SETTING] = $stopOnFailure;
+
+        return $this;
+    }
+
+    public function toPhpExpr(): Expr
+    {
+        $expr = $optionsObject = ConfigConverterTools::createObject(self::class);
+
+        // Convert recognised settings to their setter functions
+        foreach ($this->settings as $group => $groupSettings) {
+            foreach ($groupSettings as $settingName => $setting) {
+                $functionName = self::FUNCTION_NAMES_PER_SETTING[$group][$settingName] ?? null;
+                if ($settingName === self::ERROR_REPORTING_SETTING) {
+                    $setting = ConfigConverterTools::errorReportingToConstants($setting);
+                }
+                if ($functionName) {
+                    $expr = ConfigConverterTools::addMethodCall(
+                        self::class,
+                        $functionName,
+                        [$setting],
+                        $expr,
+                    );
+                    unset($this->settings[$group][$settingName]);
+                }
+            }
+        }
+
+        // Remove any now-empty groups (e.g. 'testers' => []) from the settings array
+        // Then if necessary provide any remaining settings to the object constructor. These are likely not used by
+        // Behat, but we want them in the generated config for the user to review.
+        $this->settings = array_filter($this->settings, static fn ($g) => $g !== []);
+        if ($this->settings !== []) {
+            ConfigConverterTools::addArgumentsToConstructor([$this->settings], $optionsObject);
+        }
+
+        return $expr;
+    }
+
+
+}
diff --git a/tests/Behat/Tests/Config/Converter/ConfigConverterToolsTest.php b/tests/Behat/Tests/Config/Converter/ConfigConverterToolsTest.php
new file mode 100644
index 00000000..697c0578
--- /dev/null
+++ b/tests/Behat/Tests/Config/Converter/ConfigConverterToolsTest.php
@@ -0,0 +1,250 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Behat\Tests\Config\Converter;
+
+use Behat\Config\Converter\ConfigConverterTools;
+use Behat\Config\TesterOptions;
+use PhpParser\PrettyPrinter\Standard;
+use PHPUnit\Framework\TestCase;
+
+final class ConfigConverterToolsTest extends TestCase
+{
+    public static function providerRemoveDefaultArguments(): array
+    {
+        $object = new class {
+            public function takesNoArgs(): void
+            {
+
+            }
+
+            public function withSomething(bool $something): void
+            {
+
+            }
+
+            public function withSingleDefault(bool $value = true): void
+            {
+
+            }
+
+            public function withVariadic(string ...$args): void
+            {
+
+            }
+
+            public function withMultipleArgs(string $first, string $second = 'b', string $third = 'c'): void
+            {
+
+            }
+        };
+
+        return [
+            'no args, nothing removed' => [
+                $object::class,
+                'takesNoArgs',
+                [],
+                '(new %CLASS%())->takesNoArgs()',
+            ],
+            'no default, value set false' => [
+                $object::class,
+                'withSomething',
+                [false],
+                '(new %CLASS%())->withSomething(false)',
+            ],
+            'no default, value set true' => [
+                $object::class,
+                'withSomething',
+                [true],
+                '(new %CLASS%())->withSomething(true)',
+            ],
+            'has single default, non-default value' => [
+                $object::class,
+                'withSingleDefault',
+                [false],
+                '(new %CLASS%())->withSingleDefault(false)',
+            ],
+            'has single default, default value' => [
+                $object::class,
+                'withSingleDefault',
+                [true],
+                '(new %CLASS%())->withSingleDefault()',
+            ],
+            'variadic is unchanged' => [
+                $object::class,
+                'withVariadic',
+                ['a', 'b', 'c'],
+                "(new %CLASS%())->withVariadic('a', 'b', 'c')",
+            ],
+            'multiple args, all non-default' => [
+                $object::class,
+                'withMultipleArgs',
+                ['1', '2', '3'],
+                "(new %CLASS%())->withMultipleArgs('1', '2', '3')",
+            ],
+            'multiple args, all are default' => [
+                $object::class,
+                'withMultipleArgs',
+                ['1', 'b', 'c'],
+                "(new %CLASS%())->withMultipleArgs('1')",
+            ],
+            'multiple args, last is default' => [
+                $object::class,
+                'withMultipleArgs',
+                ['1', '2', 'c'],
+                "(new %CLASS%())->withMultipleArgs('1', '2')",
+            ],
+            'multiple args, middle is default (change to named)' => [
+                $object::class,
+                'withMultipleArgs',
+                ['1', 'b', '3'],
+                "(new %CLASS%())->withMultipleArgs('1', third: '3')",
+            ],
+            'provided as named params' => [
+                $object::class,
+                'withSomething',
+                ['something' => true],
+                "(new %CLASS%())->withSomething(something: true)",
+            ],
+        ];
+    }
+
+    /**
+     * @dataProvider providerRemoveDefaultArguments
+     */
+    public function testCanRemoveDefaultArgumentsFromMethodCalls(string $class, string $method, array $args, string $expect): void
+    {
+        $expr = ConfigConverterTools::createObject($class);
+        $expr = ConfigConverterTools::addMethodCall($class, $method, $args, $expr);
+        $printer = new Standard();
+        $code = str_replace('\\' . $class, '%CLASS%', $printer->prettyPrintExpr($expr));
+        $this->assertSame($expect, $code);
+    }
+
+    public static function providerValidateMethodCallInput(): array
+    {
+        $obj = new class {
+            public function withThings(string $one, string $two): void
+            {
+
+            }
+        };
+        return [
+            'undefined class' => [
+                'This\Is\Not\A\Class',
+                'whatever',
+                [],
+                'Class "This\Is\Not\A\Class" does not exist',
+            ],
+            'undefined method' => [
+                $obj::class,
+                'withNothing',
+                [],
+                'Method class@anonymous::withNothing() does not exist',
+            ],
+            'not enough params' => [
+                $obj::class,
+                'withThings',
+                ['one'],
+                'Missing argument #1 (two)',
+            ],
+            'too many params' => [
+                $obj::class,
+                'withThings',
+                ['one', 'two', 'three'],
+                'Too many arguments provided for',
+            ],
+        ];
+    }
+
+    /**
+     * @dataProvider providerValidateMethodCallInput
+     */
+    public function testAddMethodCallValidatesInput(string $class, string $method, array $args, string $expect): void
+    {
+        $expr = ConfigConverterTools::createObject($class);
+        $this->expectExceptionMessage($expect);
+        ConfigConverterTools::addMethodCall($class, $method, $args, $expr);
+    }
+
+    public static function providerErrorReportingToConstants(): array
+    {
+        $OLD_E_STRICT = 2048;
+        $E_ALL_WITH_STRICT = E_ALL | $OLD_E_STRICT;
+
+        return [
+            [E_ALL, 'E_ALL'],
+            [0, '0'],
+            [E_ALL & ~E_DEPRECATED, 'E_ALL & ~E_DEPRECATED'],
+            [E_ALL & ~(E_DEPRECATED | E_USER_DEPRECATED), 'E_ALL & ~(E_DEPRECATED | E_USER_DEPRECATED)'],
+            [
+                E_ALL & ~(E_DEPRECATED | E_USER_DEPRECATED | E_WARNING),
+                'E_ALL & ~(E_WARNING | E_DEPRECATED | E_USER_DEPRECATED)',
+            ],
+            [
+                E_NOTICE,
+                'E_NOTICE',
+            ],
+            [
+                E_WARNING | E_NOTICE,
+                'E_WARNING | E_NOTICE',
+            ],
+            [
+                E_WARNING | E_NOTICE | E_DEPRECATED,
+                'E_WARNING | E_NOTICE | E_DEPRECATED',
+            ],
+            [
+                // Their config value includes bits that are not defined as PHP error levels.
+                E_WARNING | 65536,
+                'E_WARNING | 65536',
+            ],
+            [
+                // We can safely ignore the old E_STRICT value
+                E_ALL & ~$OLD_E_STRICT,
+                'E_ALL',
+            ],
+            [
+                // We can safely ignore the old E_STRICT value
+                $E_ALL_WITH_STRICT,
+                'E_ALL',
+            ],
+            [
+                // We can safely ignore the old E_STRICT value
+                $E_ALL_WITH_STRICT & ~E_DEPRECATED,
+                'E_ALL & ~E_DEPRECATED',
+            ],
+            [
+                // We can safely ignore the old E_STRICT value
+                $OLD_E_STRICT | E_WARNING,
+                'E_WARNING',
+            ],
+            [
+                // I guess users can still use environment vars for this...?
+                '%env(BEHAT_ERR_REPORTING)%',
+                "'%env(BEHAT_ERR_REPORTING)%'",
+            ],
+        ];
+    }
+
+    /**
+     * @dataProvider providerErrorReportingToConstants
+     */
+    public function testErrorReportingToConstants(mixed $reporting, string $expect): void
+    {
+        // Can be used with any method, but might as well test with the one that has the expected signature
+        $expr = ConfigConverterTools::createObject(TesterOptions::class);
+        $expr = ConfigConverterTools::addMethodCall(
+            TesterOptions::class,
+            'withErrorReporting',
+            [ConfigConverterTools::errorReportingToConstants($reporting)],
+            $expr,
+        );
+
+        $printer = new Standard();
+        $this->assertSame(
+            '(new \\' . TesterOptions::class . '())->withErrorReporting(' . $expect . ')',
+            $printer->prettyPrintExpr($expr),
+        );
+    }
+}
diff --git a/tests/Fixtures/ConfigStrictTesters/behat.php b/tests/Fixtures/ConfigStrictTesters/behat.php
new file mode 100644
index 00000000..06498e43
--- /dev/null
+++ b/tests/Fixtures/ConfigStrictTesters/behat.php
@@ -0,0 +1,14 @@
+<?php
+
+use Behat\Config\Config;
+use Behat\Config\Profile;
+use Behat\Config\TesterOptions;
+
+return (new Config())
+    ->withProfile(new Profile('default'))
+    ->withProfile((new Profile('with-strict'))
+        ->withTesterOptions((new TesterOptions())
+            ->withStrictResultInterpretation()))
+    ->withProfile((new Profile('not-strict'))
+        ->withTesterOptions((new TesterOptions())
+            ->withStrictResultInterpretation(false)));
diff --git a/tests/Fixtures/ConfigStrictTesters/behat.yaml b/tests/Fixtures/ConfigStrictTesters/behat.yaml
deleted file mode 100644
index bd6529a0..00000000
--- a/tests/Fixtures/ConfigStrictTesters/behat.yaml
+++ /dev/null
@@ -1,9 +0,0 @@
-default: ~
-
-with-strict:
-  testers:
-    strict: true
-
-not-strict:
-  testers:
-    strict: false
diff --git a/tests/Fixtures/ConvertConfig/full_configuration.yaml b/tests/Fixtures/ConvertConfig/full_configuration.yaml
index ecd474b6..649ec86d 100644
--- a/tests/Fixtures/ConvertConfig/full_configuration.yaml
+++ b/tests/Fixtures/ConvertConfig/full_configuration.yaml
@@ -2,6 +2,8 @@ imports:
     - imported.yaml
 
 default:
+    testers:
+        strict: true
     formatters:
         pretty:
             paths:    false
@@ -32,6 +34,8 @@ default:
         custom_extension.php: ~
 
 other:
+    calls:
+        error_reporting: 1
     formatters:
         pretty: false
 
diff --git a/tests/Fixtures/ConvertConfig/tester_options.yaml b/tests/Fixtures/ConvertConfig/tester_options.yaml
new file mode 100644
index 00000000..e35262d3
--- /dev/null
+++ b/tests/Fixtures/ConvertConfig/tester_options.yaml
@@ -0,0 +1,17 @@
+default: ~
+
+ignore-errors:
+    calls:
+        error_reporting: 22527
+
+not-strict:
+    testers:
+        strict: false
+
+complete:
+    calls:
+        error_reporting: 24565
+    testers:
+        strict: true
+        stop_on_failure: false
+        skip: true
diff --git a/tests/Fixtures/ErrorReporting/behat.php b/tests/Fixtures/ErrorReporting/behat.php
new file mode 100644
index 00000000..f70f9eec
--- /dev/null
+++ b/tests/Fixtures/ErrorReporting/behat.php
@@ -0,0 +1,14 @@
+<?php
+
+use Behat\Config\Config;
+use Behat\Config\Profile;
+use Behat\Config\TesterOptions;
+
+return (new Config())
+    ->withProfile(new Profile('default'))
+    ->withProfile((new Profile('ignore-all-but-error'))
+        ->withTesterOptions((new TesterOptions())
+            ->withErrorReporting(E_ALL & ~(E_WARNING | E_NOTICE | E_DEPRECATED))))
+    ->withProfile((new Profile('ignore-deprecations'))
+        ->withTesterOptions((new TesterOptions())
+            ->withErrorReporting(E_ALL & ~E_DEPRECATED)));
diff --git a/tests/Fixtures/ErrorReporting/behat.yaml b/tests/Fixtures/ErrorReporting/behat.yaml
deleted file mode 100644
index 62909023..00000000
--- a/tests/Fixtures/ErrorReporting/behat.yaml
+++ /dev/null
@@ -1,9 +0,0 @@
-default: ~
-
-ignore-all-but-error:
-    calls:
-        error_reporting: 24565
-
-ignore-deprecations:
-    calls:
-        error_reporting: 22527
diff --git a/tests/Fixtures/StopOnFailure/behat.php b/tests/Fixtures/StopOnFailure/behat.php
new file mode 100644
index 00000000..0d7c9ca4
--- /dev/null
+++ b/tests/Fixtures/StopOnFailure/behat.php
@@ -0,0 +1,14 @@
+<?php
+
+use Behat\Config\Config;
+use Behat\Config\Profile;
+use Behat\Config\TesterOptions;
+
+return (new Config())
+    ->withProfile(new Profile('default'))
+    ->withProfile((new Profile('stop-on-failure'))
+        ->withTesterOptions((new TesterOptions())
+            ->withStopOnFailure()))
+    ->withProfile((new Profile('no-stop-on-failure'))
+        ->withTesterOptions((new TesterOptions())
+            ->withStopOnFailure(false)));
diff --git a/tests/Fixtures/StopOnFailure/behat.yaml b/tests/Fixtures/StopOnFailure/behat.yaml
deleted file mode 100644
index 3b5d7b44..00000000
--- a/tests/Fixtures/StopOnFailure/behat.yaml
+++ /dev/null
@@ -1,9 +0,0 @@
-default: ~
-
-stop-on-failure:
-    testers:
-        stop_on_failure: true
-
-no-stop-on-failure:
-    testers:
-        stop_on_failure: false