npm package: @open-xchange/codecept-runner
Description: Parallel runner for CodeceptJS E2E tests
Latest version: 1.7.0

# @open-xchange/codecept-runner

A runner script for E2E tests. Supports parallel execution, rerunning failed tests, and continuation.

## Help

The switch `--help` (or `-h`) prints an overview with all supported switches.
Example:

```sh
pnpm codecept-runner --help
```

## CodeceptJS Configuration File

The runner needs to know the location of the CodeceptJS configuration file. If it is not located in the project root directory, the switch `--config` (or `-c`) allows to specify an alternative path.

Like CodeceptJS itself, the runner will try all well-known names for the configuration file:

- `codecept.config.js`
- `codecept.conf.js`
- `codecept.json`
- `codecept.config.ts`
- `codecept.conf.ts`

Example:

```sh
# by exact file name
pnpm codecept-runner --config=./e2e/special.config.ts

# by path with auto-detection
pnpm codecept-runner --config=./e2e
```

## Filter by Test Titles

The tests to be executed can be selected by full test titles (including scenario names and tags) using the switches `--grep` (or `-g`) and `--ignore` (or `-i`). Both switches expect a regular expression.

- If `--grep` is omitted, all existing tests will be selected.
- If `--ignore` is omitted, no selected test will be skipped.
- If `--grep` and `--ignore` are specified, the tests selected by both will be skipped.

Example:

```sh
pnpm codecept-runner --grep=@stable --ignore=@experimental
```

will execute all tests with the tag 'stable' that do not contain the tag 'experimental'.

## Retry Failed Tests

Failed tests can be rerun automatically using the switch `--retries` (or `-r`). The number after the switch specifies the number of _additional_ test runs after the initial test run.

Example:

```sh
$ pnpm codecept-runner --retries=2
running 1000 tests ...
20 tests failed

retrying 20 tests ...
2 tests failed

retrying 2 tests ...
all tests passed
```

## Parallel Execution

Tests will be executed in parallel with multiple workers.

By default, the initial number of workers is half of the number of processors available on the system. For every retry run, the number of workers will be halved to compensate for shaky tests caused by system load.

Example (on a system with 16 reported processors):

```sh
$ pnpm codecept-runner --retries=2
running 1000 tests with 8 workers ...
20 tests failed

retrying 20 tests with 4 workers ...
3 tests failed

retrying 3 tests with 2 workers ...
all tests passed
```

The number of workers can be changed using the switch `--workers` (or `-w`). The additional switch `--retry-workers` (or uppercase `-W`) allows to set a different number of workers for retrying failed tests.

Example:

```sh
$ pnpm codecept-runner --workers=4 --retries=1 --retry-workers=3
running 1000 tests with 4 workers ...
20 tests failed

retrying 20 tests with 3 workers ...
all tests passed
```

The option `--retry-workers` supports the keywords `same` and `halve` instead of a number. The keyword `same` causes to use the initial number of workers for all retry runs. The keyword `halve` (which is the default) causes to halve the number of workers for every retry run.

Example (on a system with 16 reported processors):

```sh
$ pnpm codecept-runner --retries=1 --retry-workers=same
running 1000 tests with 8 workers ...
20 tests failed

retrying 20 tests with 8 workers ...
all tests passed
```

## Partitioning

A chunk of all matching tests can be executed with the option `--partition` (or `-P`). Its value consists of the one-based chunk index and of the total number of chunks, separated with a forward slash.

```sh
$ pnpm codecept-runner --partition=2/10
found 1000 matching tests
running chunk 2 of 10 with 100 tests
all tests passed
```

## Continuation

The results of a test run will be saved in a JSON file in the configured output directory. After a test run finishes with failed tests, or has been interrupted, the remaining tests (failed and pending) can be restarted using the switch `--continue` (or uppercase `-C`).

Example:

```sh
$ pnpm codecept-runner --grep=@stable
running 1000 tests ...
20 tests failed

$ pnpm codecept-runner --continue
running 20 tests ...
all tests passed
```

The path to the JSON file can be customized with the option `--state-file` (or uppercase `-S`).

Example:

```sh
$ pnpm codecept-runner --grep=@stable --state-file=./.temp/test-state.json
running 1000 tests ...
20 tests failed

$ pnpm codecept-runner --continue --state-file=./.temp/test-state.json
running 20 tests ...
all tests passed
```

## Dry Run

Using the switch `--dry-run` (or `-d`) the runner will print log messages but will not actually run any test.

Example:

```sh
$ pnpm codecept-runner --workers=4 --retries=1 --dry-run
running 1000 tests with 4 workers ...
1000 tests skipped
```