Repository: elastic/synthetics
Pull request #254 (2021-04-14 01:17:16)

add support for custom reporters
In order to support getting access to runner events from the standard `run()` API I think it would be good to support custom `Runner` classes. This PR adds support for a `Runner` class being passed as the `reporter` option, `new`s up this class when the runner starts, and passes the `Runner` instance as well as the `ReporterOptions` to the constructor, and then moves on.

apmmachine: 
## :broken_heart: Tests Failed


<!-- BUILD BADGES-->
> _the below badges are clickable and redirect to their specific view in the CI or DOCS_
[![Pipeline View](https://img.shields.io/badge/pipeline-pipeline%20-green)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-rum%2Felastic-synthetics%2FPR-254/detail/PR-254/1//pipeline) [![Test View](https://img.shields.io/badge/test-test-green)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-rum%2Felastic-synthetics%2FPR-254/detail/PR-254/1//tests) [![Changes](https://img.shields.io/badge/changes-changes-green)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-rum%2Felastic-synthetics%2FPR-254/detail/PR-254/1//changes) [![Artifacts](https://img.shields.io/badge/artifacts-artifacts-yellow)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-rum%2Felastic-synthetics%2FPR-254/detail/PR-254/1//artifacts) [![preview](https://img.shields.io/badge/docs-preview-yellowgreen)](http://synthetics_254.docs-preview.app.elstc.co/diff)

<!-- BUILD SUMMARY-->
<details><summary>Expand to view the summary</summary>
<p>

#### Build stats


* Build Cause: `Pull request #254 opened`



* Start Time: 2021-04-14T01:17:29.780+0000


* Duration: 7 min 39 sec


* Commit: 0e91a26f34d81544eb2c33ff9f2f444407a1cc38



#### Test stats :test_tube:

| Test         | Results                         |
| ------------ | :-----------------------------: |
| Failed       | 2  |
| Passed       | 72  |
| Skipped      | 0 |
| Total        | 74   |


#### Trends :test_tube:


![Image of Build Times](https://apm-ci.elastic.co/job/apm-agent-rum/job/elastic-synthetics/job/PR-254/buildTimeGraph/png)


![Image of Tests](https://apm-ci.elastic.co/job/apm-agent-rum/job/elastic-synthetics/job/PR-254/1/testReport/history/countGraph/png?start=0&amp;end=25)


</p>
</details>

<!-- TEST RESULTS IF ANY-->

  ### Test errors [![2](https://img.shields.io/badge/2%20-red)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-rum%2Felastic-synthetics%2FPR-254/detail/PR-254/1//tests)
<details><summary>Expand to view the tests failures</summary><p>
    
    
##### `Test / network timings for aborted requests – network timings for aborted requests`


<ul>
<li>no error details</li>
<details><summary>Expand to view the stacktrace</summary><p>

```
 Error: expect(received).toBe(expected) // Object.is equality

Expected: 2
Received: 1
    at Object.<anonymous> (/var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-254/src/github.com/elastic/synthetics/__tests__/plugins/network.test.ts:122:28) 
 ```
</p></details>
</ul>
    
##### `Test / runner run - supports custom reporters – runner run - supports custom reporters`


<ul>
<li>no error details</li>
<details><summary>Expand to view the stacktrace</summary><p>

```
 Error: expect(received).toMatchInlineSnapshot(snapshot)

Snapshot name: `runner run - supports custom reporters 3`

- Snapshot  - 1
+ Received  + 1

  Object {
-   "fd": 26,
+   "fd": 22,
  }
    at Object.<anonymous> (/var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-254/src/github.com/elastic/synthetics/__tests__/core/runner.test.ts:362:30) 
 ```
</p></details>
</ul>
    
</p></details>


<!-- STEPS ERRORS IF ANY -->









### Steps errors [![1](https://img.shields.io/badge/1%20-red)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-rum%2Felastic-synthetics%2FPR-254/detail/PR-254/1//pipeline)
  <details><summary>Expand to view the steps failures</summary>
  <p>


  
  
  
  
##### `Runs the tests`
<ul>
<li>Took 0 min 27 sec . View more details on <a href="https://apm-ci.elastic.co//blue/rest/organizations/jenkins/pipelines/apm-agent-rum/pipelines/elastic-synthetics/pipelines/PR-254/runs/1/steps/300/log/?start=0">here</a></li>
<li>Description: <code>./node_modules/.bin/jest --ci --reporters=default --reporters=jest-junit</code></l1>
</ul>
  
  </p>
  </details>



<!-- LOG OUTPUT-->
### Log output

<details><summary>Expand to view the last 100 lines of log output</summary>
<p>

```
[2021-04-14T01:23:36.509Z] + ./node_modules/.bin/jest --ci --reporters=default --reporters=jest-junit
[2021-04-14T01:23:39.810Z] 
[2021-04-14T01:23:39.810Z] Running without BrowserService
[2021-04-14T01:23:49.823Z] PASS __tests__/helper.test.ts (8.115 s)
[2021-04-14T01:23:50.395Z] PASS __tests__/reporters/json.test.ts
[2021-04-14T01:23:51.788Z] PASS __tests__/reporters/junit.test.ts
[2021-04-14T01:23:53.184Z] FAIL __tests__/plugins/network.test.ts (13.008 s)
[2021-04-14T01:23:53.184Z]   ● network › timings for aborted requests
[2021-04-14T01:23:53.184Z] 
[2021-04-14T01:23:53.184Z]     expect(received).toBe(expected) // Object.is equality
[2021-04-14T01:23:53.184Z] 
[2021-04-14T01:23:53.184Z]     Expected: 2
[2021-04-14T01:23:53.184Z]     Received: 1
[2021-04-14T01:23:53.184Z] 
[2021-04-14T01:23:53.184Z]       120 |     await Gatherer.dispose(driver);
[2021-04-14T01:23:53.184Z]       121 |     const netinfo = await network.stop();
[2021-04-14T01:23:53.184Z]     > 122 |     expect(netinfo.length).toBe(2);
[2021-04-14T01:23:53.184Z]           |                            ^
[2021-04-14T01:23:53.184Z]       123 |     expect(netinfo[1]).toMatchObject({
[2021-04-14T01:23:53.184Z]       124 |       url: `${server.PREFIX}/delay100`,
[2021-04-14T01:23:53.184Z]       125 |       status: 0,
[2021-04-14T01:23:53.184Z] 
[2021-04-14T01:23:53.184Z]       at Object.<anonymous> (__tests__/plugins/network.test.ts:122:28)
[2021-04-14T01:23:53.184Z] 
[2021-04-14T01:23:54.576Z] PASS __tests__/core/index.test.ts
[2021-04-14T01:23:55.528Z] PASS __tests__/reporters/base.test.ts
[2021-04-14T01:23:57.453Z] PASS __tests__/cli.test.ts (5.414 s)
[2021-04-14T01:23:58.025Z] PASS __tests__/index.test.ts
[2021-04-14T01:23:58.968Z] FAIL __tests__/core/runner.test.ts (18.84 s)
[2021-04-14T01:23:58.968Z]   ● runner › run - supports custom reporters
[2021-04-14T01:23:58.968Z] 
[2021-04-14T01:23:58.968Z]     expect(received).toMatchInlineSnapshot(snapshot)
[2021-04-14T01:23:58.968Z] 
[2021-04-14T01:23:58.968Z]     Snapshot name: `runner run - supports custom reporters 3`
[2021-04-14T01:23:58.968Z] 
[2021-04-14T01:23:58.968Z]     - Snapshot  - 1
[2021-04-14T01:23:58.968Z]     + Received  + 1
[2021-04-14T01:23:58.968Z] 
[2021-04-14T01:23:58.968Z]       Object {
[2021-04-14T01:23:58.968Z]     -   "fd": 26,
[2021-04-14T01:23:58.968Z]     +   "fd": 22,
[2021-04-14T01:23:58.968Z]       }
[2021-04-14T01:23:58.968Z] 
[2021-04-14T01:23:58.968Z]       360 |     `);
[2021-04-14T01:23:58.968Z]       361 |     expect(reporter.runner).toBeInstanceOf(Runner);
[2021-04-14T01:23:58.968Z]     > 362 |     expect(reporter.options).toMatchInlineSnapshot(`
[2021-04-14T01:23:58.968Z]           |                              ^
[2021-04-14T01:23:58.968Z]       363 |       Object {
[2021-04-14T01:23:58.968Z]       364 |         "fd": 26,
[2021-04-14T01:23:58.968Z]       365 |       }
[2021-04-14T01:23:58.968Z] 
[2021-04-14T01:23:58.968Z]       at Object.<anonymous> (__tests__/core/runner.test.ts:362:30)
[2021-04-14T01:23:58.968Z] 
[2021-04-14T01:23:58.968Z]  › 1 snapshot failed.
[2021-04-14T01:23:59.230Z] PASS __tests__/plugins/browser-console.test.ts
[2021-04-14T01:24:00.687Z] PASS __tests__/plugins/performance.test.ts
[2021-04-14T01:24:00.687Z] PASS __tests__/plugins/plugin-manager.test.ts
[2021-04-14T01:24:01.262Z] PASS __tests__/plugins/tracing.test.ts
[2021-04-14T01:24:02.209Z] PASS __tests__/core/logger.test.ts
[2021-04-14T01:24:02.791Z] PASS __tests__/core/browser-service.test.ts
[2021-04-14T01:24:03.052Z] PASS __tests__/dsl/journey.test.ts
[2021-04-14T01:24:03.052Z] PASS __tests__/core/gatherer.test.ts
[2021-04-14T01:24:03.052Z] 
[2021-04-14T01:24:03.052Z] Snapshot Summary
[2021-04-14T01:24:03.052Z]  › 1 snapshot failed from 1 test suite. Inspect your code changes or re-run jest with `-u` to update them.
[2021-04-14T01:24:03.053Z] 
[2021-04-14T01:24:03.053Z] Test Suites: 2 failed, 15 passed, 17 total
[2021-04-14T01:24:03.053Z] Tests:       2 failed, 58 passed, 60 total
[2021-04-14T01:24:03.053Z] Snapshots:   1 failed, 10 passed, 11 total
[2021-04-14T01:24:03.053Z] Time:        23.34 s
[2021-04-14T01:24:03.053Z] Ran all test suites.
[2021-04-14T01:24:05.703Z] Post stage
[2021-04-14T01:24:05.732Z] Recording test results
[2021-04-14T01:24:05.777Z] [Checks API] No suitable checks publisher found.
[2021-04-14T01:24:05.937Z] Stage "E2e Test" skipped due to earlier failure(s)
[2021-04-14T01:24:06.032Z] Stage "Publish Docker image" skipped due to earlier failure(s)
[2021-04-14T01:24:06.176Z] Stage "Matrix - STACK_VERSION = '8.0.0-synthetics'" skipped due to earlier failure(s)
[2021-04-14T01:24:06.185Z] Stage "Matrix - STACK_VERSION = '7.10.0-synthetics'" skipped due to earlier failure(s)
[2021-04-14T01:24:06.194Z] Stage "Matrix - STACK_VERSION = '7.11.0-synthetics'" skipped due to earlier failure(s)
[2021-04-14T01:24:06.276Z] Stage "Matrix - STACK_VERSION = '8.0.0-synthetics'" skipped due to earlier failure(s)
[2021-04-14T01:24:06.289Z] Stage "Matrix - STACK_VERSION = '7.10.0-synthetics'" skipped due to earlier failure(s)
[2021-04-14T01:24:06.298Z] Stage "Matrix - STACK_VERSION = '7.11.0-synthetics'" skipped due to earlier failure(s)
[2021-04-14T01:24:06.487Z] Stage "Matrix - STACK_VERSION = '8.0.0-synthetics'" skipped due to earlier failure(s)
[2021-04-14T01:24:06.500Z] Stage "Matrix - STACK_VERSION = '7.10.0-synthetics'" skipped due to earlier failure(s)
[2021-04-14T01:24:06.508Z] Stage "Matrix - STACK_VERSION = '7.11.0-synthetics'" skipped due to earlier failure(s)
[2021-04-14T01:24:06.705Z] Stage "Matrix - STACK_VERSION = '8.0.0-synthetics'" skipped due to earlier failure(s)
[2021-04-14T01:24:06.717Z] Stage "Matrix - STACK_VERSION = '7.10.0-synthetics'" skipped due to earlier failure(s)
[2021-04-14T01:24:06.725Z] Stage "Matrix - STACK_VERSION = '7.11.0-synthetics'" skipped due to earlier failure(s)
[2021-04-14T01:24:06.968Z] Failed in branch Matrix - STACK_VERSION = '8.0.0-synthetics'
[2021-04-14T01:24:06.980Z] Failed in branch Matrix - STACK_VERSION = '7.10.0-synthetics'
[2021-04-14T01:24:06.993Z] Failed in branch Matrix - STACK_VERSION = '7.11.0-synthetics'
[2021-04-14T01:24:07.188Z] Stage "Release" skipped due to earlier failure(s)
[2021-04-14T01:24:07.236Z] Stage "Release" skipped due to earlier failure(s)
[2021-04-14T01:24:07.332Z] Stage "Release" skipped due to earlier failure(s)
[2021-04-14T01:24:07.698Z] Running on Jenkins in /var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-254
[2021-04-14T01:24:07.796Z] [INFO] getVaultSecret: Getting secrets
[2021-04-14T01:24:08.059Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-04-14T01:24:09.133Z] + chmod 755 generate-build-data.sh
[2021-04-14T01:24:09.133Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-rum/elastic-synthetics/PR-254/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-rum/elastic-synthetics/PR-254/runs/1 FAILURE 399077
[2021-04-14T01:24:09.384Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-rum/elastic-synthetics/PR-254/runs/1/steps/?limit=10000 -o steps-info.json

```

</p>
</details>



<!--COMMENT_GENERATED_WITH_ID_comment.id-->

Patch:
diff --git a/src/core/runner.ts b/src/core/runner.ts
index 2ff76a7..bfd338f 100644
--- a/src/core/runner.ts
+++ b/src/core/runner.ts
@@ -26,7 +26,7 @@
 import { EventEmitter } from 'events';
 import { Journey } from '../dsl/journey';
 import { Step } from '../dsl/step';
-import { reporters } from '../reporters';
+import { reporters, Reporter } from '../reporters';
 import { getMonotonicTime, getTimestamp, runParallel } from '../helpers';
 import {
   StatusValue,
@@ -42,9 +42,16 @@ import { log } from './logger';
 
 export type RunOptions = Omit<
   CliArgs,
-  'debug' | 'json' | 'pattern' | 'inline' | 'require' | 'suiteParams'
+  | 'debug'
+  | 'json'
+  | 'pattern'
+  | 'inline'
+  | 'require'
+  | 'suiteParams'
+  | 'reporter'
 > & {
   params?: RunParamaters;
+  reporter?: CliArgs['reporter'] | Reporter;
 };
 
 type RunParamaters = Record<string, unknown>;
@@ -301,7 +308,10 @@ export default class Runner {
     /**
      * Set up the corresponding reporter and fallback
      */
-    const Reporter = reporters[reporter] || reporters['default'];
+    const Reporter =
+      typeof reporter === 'function'
+        ? reporter
+        : reporters[reporter] || reporters['default'];
     new Reporter(this, { fd: outfd });
     this.emit('start', { numJourneys: this.journeys.length });
     await this.runBeforeAllHook();
diff --git a/src/index.ts b/src/index.ts
index b5037e4..1f4776a 100644
--- a/src/index.ts
+++ b/src/index.ts
@@ -65,3 +65,9 @@ export type {
   ChromiumBrowserContext,
   CDPSession,
 } from 'playwright-chromium';
+
+/**
+ * Export the types necessary to write custom reporters
+ */
+export type { default as Runner } from './core/runner';
+export type { Reporter, ReporterOptions } from './reporters';
diff --git a/src/reporters/base.ts b/src/reporters/base.ts
index 8a064c5..a728190 100644
--- a/src/reporters/base.ts
+++ b/src/reporters/base.ts
@@ -33,11 +33,7 @@ import {
   findPWLogsIndexes,
   rewriteErrorStack,
 } from '../helpers';
-
-export type ReporterOptions = {
-  fd?: number;
-  colors?: boolean;
-};
+import { ReporterOptions } from './reporter';
 
 function renderError(error) {
   let output = '';
diff --git a/src/reporters/index.ts b/src/reporters/index.ts
index 0e7d967..7f5cc0b 100644
--- a/src/reporters/index.ts
+++ b/src/reporters/index.ts
@@ -26,7 +26,9 @@
 import BaseReporter from './base';
 import JSONReporter from './json';
 import JUnitReporter from './junit';
+import Reporter, { ReporterOptions } from './reporter';
 
+export { Reporter, ReporterOptions };
 export const reporters = {
   default: BaseReporter,
   json: JSONReporter,
diff --git a/src/reporters/reporter.ts b/src/reporters/reporter.ts
new file mode 100644
index 0000000..dbe5c7b
--- /dev/null
+++ b/src/reporters/reporter.ts
@@ -0,0 +1,35 @@
+/**
+ * MIT License
+ *
+ * Copyright (c) 2020-present, Elastic NV
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ *
+ */
+
+import Runner from '../core/runner';
+
+export type ReporterOptions = {
+  fd?: number;
+  colors?: boolean;
+};
+
+export default interface Reporter {
+  new (runner: Runner, options: ReporterOptions): any;
+}


Test patch:
diff --git a/__tests__/core/runner.test.ts b/__tests__/core/runner.test.ts
index 9d6cbf8..49ce1ff 100644
--- a/__tests__/core/runner.test.ts
+++ b/__tests__/core/runner.test.ts
@@ -309,4 +309,54 @@ describe('runner', () => {
       'afterAll2',
     ]);
   });
+
+  it('run - supports custom reporters', async () => {
+    let reporter;
+    class Reporter {
+      messages: string[] = [];
+
+      constructor(
+        public readonly runner: Runner,
+        public readonly options: any
+      ) {
+        reporter = this;
+
+        this.runner.on('start', ({ numJourneys }) => {
+          this.messages.push(`numJourneys ${numJourneys}`);
+        });
+        this.runner.on('journey:start', ({ journey }) => {
+          this.messages.push(`journey:start ${journey.name}`);
+        });
+        this.runner.on('journey:end', ({ journey }) => {
+          this.messages.push(`journey:end ${journey.name}`);
+        });
+        this.runner.on('end', () => {
+          this.messages.push(`end`);
+        });
+      }
+    }
+
+    runner.addJourney(new Journey({ name: 'foo' }, noop));
+    const result = await runner.run({
+      reporter: Reporter,
+      wsEndpoint,
+      outfd: fs.openSync(dest, 'w'),
+    });
+
+    expect(result).toEqual({
+      foo: {
+        status: 'succeeded',
+      },
+    });
+    expect(reporter?.messages).toEqual([
+      'numJourneys 1',
+      'journey:start foo',
+      'journey:end foo',
+      'end',
+    ]);
+    expect(reporter.runner).toBeInstanceOf(Runner);
+    expect(reporter.options).toEqual({
+      fd: expect.any(Number),
+    });
+  });
 });