npm package: @pumpo5.dev/pumpo-reporter
Description: Pumpo no.5's own custom reporter, based on Playwright's default 'line' reporter
Author: Alvaro A. de Souza Silva
Latest version: 0.0.133

# Playwright Prometheus Reporter

> This is a custom reporter for Playwright.
> 
> Use it for collecting metrics when running a suite of tests written in Playwright, and push those metrics to a Prometheus Pushgateway.
> 
> The metrics can then be scrapped by Prometheus and used as you want, for example, in Grafana visualizations.

## Getting Started

Assuming that you already have one or more Playwright test suites ready in your system, all you need to do is:

* Install this package in your project.
* Add this package name to the list of reporters in your Playwright configuration file.

## Installation

Install this package in your project which includes the Playwright tests:

```sh
$ npm install --save-dev @pumpo5.dev/playwright-prometheus-reporter
```

Or if you prefer Yarn:

```sh
$ yarn add --dev @pumpo5.dev/playwright-prometheus-reporter
```

Add this package to the list of reporters in your Playwright configuration (usually in a **playwright.config.ts** file):

```sh
...

reporter: "@pumpo5.dev/playwright-prometheus-reporter",

...
```

For more information on Playwright reporters, and other ways to add a new reporter to your test suite, please read [the official documentation](https://playwright.dev/docs/test-reporters).

## Usage

In order to enable the reporter and start collecting metrics, you need to create a configuration file. **This file must be named _config.conf_**.

Before any tests are executed, the reporter will load and look for this file in three ways:

* First, it will check if an environment variable named **CONFIG_PATH** is set. 
    
  If it is, it will then check its value and verify if it is a path that points to a file named **config.conf**.

  In case positive, it will then try to load the configuration by reading this file.


* Second, if no **CONFIG_PATH** environment variable is set, the reporter will check if a command line argument was passed to NodeJS contaning a path to any file named **config.conf**.
  In case positive, it will then try to load the configuration by reading the file pointed by this command line argument.


* Third and last, in case the other methods failed, the reporter will look for a **config.conf** file  in the project's root folder and directories above.

  In case it finds a **config.conf** in any of those directories it will proceed to load it and use it to set its internal configuration.

The keys that could or should be included in this configuration file are as follows:

Key | Type    | Mandatory | Description
---|---------|---|---
pn5.reporting.enabled | boolean | Yes | Enables the Prometheus reporter.
pn5.reporting.prometheus.enabled | boolean | Yes | Enables the collection and pushing of metrics during tests.
pn5.reporting.prometheus.endpoint | String | Yes | URL for the Pushgateway endpoint to which metrics should be pushed.
pn5.reporting.prometheus.suite | String | Yes | Name to be used for the **suite_name** label in metrics. In case this is empty, some class name will be used as default, which can be confusing.
pn5.reporting.prometheus.testcases.track | boolean | No| Enables metrics for test cases. If disabled, only suite metrics will be collected.
pn5.reporting.prometheus.suite_url | String | No | URL to be used as a value for the **suite_url** label in metrics.
pn5.reporting.prometheus.branch | String | No | The current development branch in the versioning system. Used as a value for the **branch** label in metrics.
pn5.reporting.prometheus.job | String | No | A name to be given for the job when pushing metrics, and for the **job** label. If empty, the suite name will be used as default. **Warning: the slash '/' and backslash '\' characters is not allowed in the job name. In case it is present, the Pushgateway will not accept the metrics.**
pn5.reporting.prometheus.project | String | No | The project name, to be prepended to the jobname
pn5.reporting.prometheus.labels | String | No| Custom labels and values to be pushed in metrics, together with the base labels listed in the tables above. Should be in format **label1=value1,label2=value2,...**.
pn5.reporting.prometheus.username | String | No | The user name, in case the endpoint requires basic authentication
pn5.reporting.prometheus.password | String | No | The password, in case the endpoint requires basic authentication

The metrics currently reported are:

#### Suite metrics

Metric Name | Description                                                                                                                    | Base Labels
---|--------------------------------------------------------------------------------------------------------------------------------|---
suite_execution_total | Total number of tests in the test suite.                                                                                       | **suite_name**: a specific identifier for the test suite<br>**suite_url**: any URL the user wants the test suite to point to, e.g., a CI/CD pipeline<br>**branch**: the branch in the version control system from where the tests were run
suite_execution_failed | Total number of failing tests in the test suite. This includes tests with status **failed**, **timedOut** and **interrupted**. | Same as above.
suite_execution_finish_time | Timestamp from when the last execution finished (in seconds).                                                                  | Same as above.
suite_execution_duration_seconds | How long the suite took to finish, in seconds.                                                                                 | Same as above.

#### Test case metrics

Metric Name | Description                                                                                                   | Base Labels
---|---------------------------------------------------------------------------------------------------------------|---
test_execution_duration_seconds | How long the test case took to finish, in seconds.                                                            | Same as above, with the addition of:<br><br>**test_case_name**: the name of the test case.
test_execution_finish_time | Timestamp from when the last test case execution finished (in seconds).                                       | Same as above.
test_execution_total | Displays a test case execution result, in a label. The metric value is the order in which the tests finished. | Same as above, with the addition of:<br><br>**result**: what was the outcome from test case. Possible values are either **passed, failed, timedOut** or **interrupted**.

## Examples

Please, check our repository to see example Playwright test suites using the Playwright Prometheus Reporter [here](https://dev.azure.com/pumpitup/pumpo-number-five/_git/pumpo-number-five-nodejs?path=/clients/integration-tests/reporting/prometheus/playwright-tests).

## Other versions

If you like this project, please check **Pumpo#5 in Java** as well, including another Prometheus reporter, [here](https://pumpo5.dev/#/) and [here](https://dev.azure.com/pumpitup/pumpo-number-five/_git/pumpo-number-five) :)

## Authors

* **Alvaro  Silva** - *Initial work* - [Linkedin](https://www.linkedin.com/in/alvarosouzasilva/)

## License

[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) © [PumpITup](https://pumpitup.cz)