Repository: elastic/synthetics
Pull request #357 (2021-07-28 03:11:52)

Add --quiet-exit-code and make it default --rich-events behavior
Fixes https://github.com/elastic/synthetics/issues/356

Accomplishes the fix by making --rich-events always exit with a code of 0, even if tests fail

apmmachine: 
## :broken_heart: Tests Failed


<!-- BUILD BADGES-->
> _the below badges are clickable and redirect to their specific view in the CI or DOCS_
[![Pipeline View](https://img.shields.io/badge/pipeline-pipeline%20-green)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-rum%2Felastic-synthetics%2FPR-357/detail/PR-357/1//pipeline) [![Test View](https://img.shields.io/badge/test-test-green)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-rum%2Felastic-synthetics%2FPR-357/detail/PR-357/1//tests) [![Changes](https://img.shields.io/badge/changes-changes-green)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-rum%2Felastic-synthetics%2FPR-357/detail/PR-357/1//changes) [![Artifacts](https://img.shields.io/badge/artifacts-artifacts-yellow)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-rum%2Felastic-synthetics%2FPR-357/detail/PR-357/1//artifacts) [![preview](https://img.shields.io/badge/docs-preview-yellowgreen)](http://synthetics_357.docs-preview.app.elstc.co/diff)  [![preview](https://img.shields.io/badge/elastic-observability-blue)](https://ci-stats.elastic.co/app/apm/services/apm-ci/transactions/view?rangeFrom=2021-07-28T03:02:06.632Z&rangeTo=2021-07-28T03:22:06.632Z&transactionName=apm-agent-rum/elastic-synthetics/PR-{number}&transactionType=unknown&latencyAggregationType=avg&traceId=006ab7188ea6f08e64fb8f224b8b74f9&transactionId=f4a0d1b77fe8236d)

<!-- BUILD SUMMARY-->
<details><summary>Expand to view the summary</summary>
<p>

#### Build stats




* Start Time: 2021-07-28T03:12:06.632+0000


* Duration: 6 min 44 sec


* Commit: 964ff8ba8ffff16c068f31227605ce7b433bc368



#### Test stats :test_tube:

| Test         | Results                         |
| ------------ | :-----------------------------: |
| Failed       | 1  |
| Passed       | 118  |
| Skipped      | 0 |
| Total        | 119   |


#### Trends :test_tube:


![Image of Build Times](https://apm-ci.elastic.co/job/apm-agent-rum/job/elastic-synthetics/job/PR-357/buildTimeGraph/png)


![Image of Tests](https://apm-ci.elastic.co/job/apm-agent-rum/job/elastic-synthetics/job/PR-357/1/testReport/history/countGraph/png?start=0&amp;end=25)


</p>
</details>

<!-- TEST RESULTS IF ANY-->

  ### Test errors [![1](https://img.shields.io/badge/1%20-red)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-rum%2Felastic-synthetics%2FPR-357/detail/PR-357/1//tests)
<details><summary>Expand to view the tests failures</summary><p>
    
    
##### `Test / CLI mimick new heartbeat with `--rich-events` flag – CLI mimick new heartbeat with `--rich-events` flag`


<ul>
<li>no error details</li>
<details><summary>Expand to view the stacktrace</summary><p>

```
 : Timeout - Async callback was not invoked within the 5000 ms timeout specified by jest.setTimeout.Timeout - Async callback was not invoked within the 5000 ms timeout specified by jest.setTimeout.Error: 
    at new Spec (/var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-357/src/github.com/elastic/synthetics/node_modules/jest-jasmine2/build/jasmine/Spec.js:116:22)
    at new Spec (/var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-357/src/github.com/elastic/synthetics/node_modules/jest-jasmine2/build/setup_jest_globals.js:78:9)
    at specFactory (/var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-357/src/github.com/elastic/synthetics/node_modules/jest-jasmine2/build/jasmine/Env.js:523:24)
    at Env.it (/var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-357/src/github.com/elastic/synthetics/node_modules/jest-jasmine2/build/jasmine/Env.js:592:24)
    at Env.it (/var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-357/src/github.com/elastic/synthetics/node_modules/jest-jasmine2/build/jasmineAsyncInstall.js:134:23)
    at it (/var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-357/src/github.com/elastic/synthetics/node_modules/jest-jasmine2/build/jasmine/jasmineLight.js:100:21)
    at Suite.<anonymous> (/var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-357/src/github.com/elastic/synthetics/__tests__/cli.test.ts:88:3)
    at addSpecsToSuite (/var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-357/src/github.com/elastic/synthetics/node_modules/jest-jasmine2/build/jasmine/Env.js:444:51)
    at Env.describe (/var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-357/src/github.com/elastic/synthetics/node_modules/jest-jasmine2/build/jasmine/Env.js:414:11)
    at describe (/var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-357/src/github.com/elastic/synthetics/node_modules/jest-jasmine2/build/jasmine/jasmineLight.js:88:18) 
 ```
</p></details>
</ul>
    
</p></details>


<!-- STEPS ERRORS IF ANY -->









### Steps errors [![1](https://img.shields.io/badge/1%20-red)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-rum%2Felastic-synthetics%2FPR-357/detail/PR-357/1//pipeline)
  <details><summary>Expand to view the steps failures</summary>
  <p>


  
  
  
  
##### `Runs the tests`
<ul>
<li>Took 1 min 56 sec . View more details on <a href="https://apm-ci.elastic.co//blue/rest/organizations/jenkins/pipelines/apm-agent-rum/pipelines/elastic-synthetics/pipelines/PR-357/runs/1/steps/300/log/?start=0">here</a></li>
<li>Description: <code>./node_modules/.bin/jest --ci --reporters=default --reporters=jest-junit</code></l1>
</ul>
  
  </p>
  </details>



<!-- LOG OUTPUT-->
### Log output

<details><summary>Expand to view the last 100 lines of log output</summary>
<p>

```
[2021-07-28T03:16:19.359Z] + npm run build
[2021-07-28T03:16:19.359Z] 
[2021-07-28T03:16:19.359Z] > @elastic/synthetics@1.0.0-beta.10 build /var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-357/src/github.com/elastic/synthetics
[2021-07-28T03:16:19.359Z] > tsc
[2021-07-28T03:16:19.359Z] 
[2021-07-28T03:16:23.876Z] Timeout set to expire in 5 min 0 sec
[2021-07-28T03:16:24.905Z] Running in /var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-357/src/github.com/elastic/synthetics
[2021-07-28T03:16:39.735Z] + set +x
[2021-07-28T03:16:39.735Z] => nvm is already installed in /var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-357/.nvm, trying to update using git
[2021-07-28T03:16:40.309Z] => => Compressing and cleaning up git repository
[2021-07-28T03:16:40.309Z] 
[2021-07-28T03:16:40.309Z] => Profile not found. Tried ~/.bashrc, ~/.bash_profile, ~/.zshrc, and ~/.profile.
[2021-07-28T03:16:40.309Z] => Create one of them and run this script again
[2021-07-28T03:16:40.309Z]    OR
[2021-07-28T03:16:40.309Z] => Append the following lines to the correct file yourself:
[2021-07-28T03:16:40.309Z] 
[2021-07-28T03:16:40.309Z] export NVM_DIR="$HOME/.nvm"
[2021-07-28T03:16:40.309Z] [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
[2021-07-28T03:16:40.309Z] 
[2021-07-28T03:16:42.224Z] => Close and reopen your terminal to start using nvm or run the following to use it now:
[2021-07-28T03:16:42.224Z] 
[2021-07-28T03:16:42.224Z] export NVM_DIR="$HOME/.nvm"
[2021-07-28T03:16:42.224Z] [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
[2021-07-28T03:16:42.224Z] Found '/var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-357/src/github.com/elastic/synthetics/.nvmrc' with version <12.22.3>
[2021-07-28T03:16:42.224Z] v12.22.3 is already installed.
[2021-07-28T03:16:42.493Z] Now using node v12.22.3 (npm v6.14.13)
[2021-07-28T03:16:42.494Z] /var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-357
[2021-07-28T03:16:42.682Z] Running in /var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-357/src/github.com/elastic/synthetics
[2021-07-28T03:16:43.036Z] + npm add --dev jest-junit
[2021-07-28T03:16:43.302Z] npm WARN install Usage of the `--dev` option is deprecated. Use `--also=dev` instead.
[2021-07-28T03:16:49.927Z] npm WARN @elastic/synthetics@1.0.0-beta.10 No repository field.
[2021-07-28T03:16:49.927Z] npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.3.2 (node_modules/fsevents):
[2021-07-28T03:16:49.927Z] npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.3.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
[2021-07-28T03:16:49.927Z] 
[2021-07-28T03:16:49.927Z] + jest-junit@12.2.0
[2021-07-28T03:16:49.927Z] added 4 packages from 9 contributors and audited 789 packages in 5.748s
[2021-07-28T03:16:49.927Z] 
[2021-07-28T03:16:49.927Z] 68 packages are looking for funding
[2021-07-28T03:16:49.927Z]   run `npm fund` for details
[2021-07-28T03:16:49.927Z] 
[2021-07-28T03:16:49.927Z] found 10 vulnerabilities (9 moderate, 1 high)
[2021-07-28T03:16:49.927Z]   run `npm audit fix` to fix them, or `npm audit` for details
[2021-07-28T03:16:50.296Z] + ./node_modules/.bin/jest --ci --reporters=default --reporters=jest-junit
[2021-07-28T03:16:54.225Z] 
[2021-07-28T03:16:54.225Z] Running without BrowserService
[2021-07-28T03:17:12.350Z] PASS __tests__/reporters/json.test.ts (17.815 s)
[2021-07-28T03:17:20.545Z] PASS __tests__/plugins/network.test.ts (6.925 s)
[2021-07-28T03:17:20.813Z] PASS __tests__/sdk/trace-metrics.test.ts
[2021-07-28T03:17:22.234Z] PASS __tests__/helper.test.ts
[2021-07-28T03:17:22.234Z] PASS __tests__/core/runner.test.ts (27.865 s)
[2021-07-28T03:17:24.181Z] PASS __tests__/reporters/junit.test.ts
[2021-07-28T03:17:24.181Z] PASS __tests__/reporters/base.test.ts
[2021-07-28T03:17:26.109Z] PASS __tests__/core/index.test.ts
[2021-07-28T03:17:27.055Z] PASS __tests__/index.test.ts
[2021-07-28T03:17:28.476Z] PASS __tests__/plugins/browser-console.test.ts
[2021-07-28T03:17:29.441Z] PASS __tests__/plugins/tracing.test.ts
[2021-07-28T03:17:32.675Z] PASS __tests__/plugins/performance.test.ts
[2021-07-28T03:17:32.675Z] PASS __tests__/core/gatherer.test.ts
[2021-07-28T03:17:33.646Z] PASS __tests__/plugins/plugin-manager.test.ts
[2021-07-28T03:17:33.912Z] PASS __tests__/config.test.ts
[2021-07-28T03:17:35.378Z] PASS __tests__/dsl/journey.test.ts
[2021-07-28T03:17:35.951Z] PASS __tests__/core/browser-service.test.ts
[2021-07-28T03:17:36.212Z] PASS __tests__/core/logger.test.ts
[2021-07-28T03:17:37.158Z] PASS __tests__/sdk/trace-processor.test.ts
[2021-07-28T03:17:45.302Z] FAIL __tests__/cli.test.ts (50.915 s)
[2021-07-28T03:17:45.302Z]   ● CLI › mimick new heartbeat with `--rich-events` flag
[2021-07-28T03:17:45.302Z] 
[2021-07-28T03:17:45.302Z]     : Timeout - Async callback was not invoked within the 5000 ms timeout specified by jest.setTimeout.Timeout - Async callback was not invoked within the 5000 ms timeout specified by jest.setTimeout.Error:
[2021-07-28T03:17:45.302Z] 
[2021-07-28T03:17:45.302Z]       86 |   });
[2021-07-28T03:17:45.302Z]       87 | 
[2021-07-28T03:17:45.302Z]     > 88 |   it('mimick new heartbeat with `--rich-events` flag', async () => {
[2021-07-28T03:17:45.302Z]          |   ^
[2021-07-28T03:17:45.302Z]       89 |     const cli = new CLIMock([
[2021-07-28T03:17:45.302Z]       90 |       join(FIXTURES_DIR, 'fake.journey.ts'),
[2021-07-28T03:17:45.302Z]       91 |       join(FIXTURES_DIR, 'error.journey.ts'),
[2021-07-28T03:17:45.302Z] 
[2021-07-28T03:17:45.302Z]       at new Spec (node_modules/jest-jasmine2/build/jasmine/Spec.js:116:22)
[2021-07-28T03:17:45.302Z]       at Suite.<anonymous> (__tests__/cli.test.ts:88:3)
[2021-07-28T03:17:45.302Z] 
[2021-07-28T03:17:45.302Z] Test Suites: 1 failed, 19 passed, 20 total
[2021-07-28T03:17:45.302Z] Tests:       1 failed, 104 passed, 105 total
[2021-07-28T03:17:45.302Z] Snapshots:   11 passed, 11 total
[2021-07-28T03:17:45.302Z] Time:        51.386 s
[2021-07-28T03:17:45.302Z] Ran all test suites.
[2021-07-28T03:17:46.699Z] Post stage
[2021-07-28T03:17:46.737Z] Recording test results
[2021-07-28T03:17:46.792Z] [Checks API] No suitable checks publisher found.
[2021-07-28T03:17:46.993Z] Stage "E2e Test" skipped due to earlier failure(s)
[2021-07-28T03:17:47.104Z] Stage "Release" skipped due to earlier failure(s)
[2021-07-28T03:17:47.157Z] Stage "Release" skipped due to earlier failure(s)
[2021-07-28T03:17:47.264Z] Stage "Release" skipped due to earlier failure(s)
[2021-07-28T03:17:49.241Z] Running on worker-1095690 in /var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-357
[2021-07-28T03:17:49.330Z] [INFO] getVaultSecret: Getting secrets
[2021-07-28T03:17:49.393Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-07-28T03:17:51.717Z] + chmod 755 generate-build-data.sh
[2021-07-28T03:17:51.717Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-rum/elastic-synthetics/PR-357/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-rum/elastic-synthetics/PR-357/runs/1 FAILURE 343607
[2021-07-28T03:17:51.717Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-rum/elastic-synthetics/PR-357/runs/1/steps/?limit=10000 -o steps-info.json
[2021-07-28T03:17:53.811Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-rum/elastic-synthetics/PR-357/runs/1/tests/?status=FAILED -o tests-errors.json
[2021-07-28T03:17:54.511Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-agent-rum/elastic-synthetics/PR-357/runs/1/log/ -o pipeline-log.txt

```

</p>
</details>



<!--COMMENT_GENERATED_WITH_ID_comment.id-->

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 22cdff3..60e8e32 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,15 @@
 
 All notable changes to this project will be documented in this file.
 
+## v1.0.0-beta.10 (2021-07-27)
+
+### Bug fixes
+
+- Record correct screenshots for popups in new tabs
+  and windows [#353](https://github.com/elastic/synthetics/pull/353)
+- Capture URL correctly for steps that involve navigations inside
+  popups and windows [#352](https://github.com/elastic/synthetics/pull/352)
+
 ## v1.0.0-beta.9 (2021-07-13)
 
 ### Bug fixes
diff --git a/src/cli.ts b/src/cli.ts
index e31c8f6..f19a692 100644
--- a/src/cli.ts
+++ b/src/cli.ts
@@ -194,12 +194,14 @@ async function prepareSuites(inputs: string[]) {
     ...options,
   });
 
-  /**
-   * Exit with error status if any journey fails
-   */
-  for (const result of Object.values(results)) {
-    if (result.status === 'failed') {
-      process.exit(1);
+  if (!options.quietExitCode) {
+    /**
+     * Exit with error status if any journey fails
+     */
+    for (const result of Object.values(results)) {
+      if (result.status === 'failed') {
+        process.exit(1);
+      }
     }
   }
 })().catch(e => {
diff --git a/src/common_types.ts b/src/common_types.ts
index 950c096..c096354 100644
--- a/src/common_types.ts
+++ b/src/common_types.ts
@@ -130,6 +130,7 @@ export type CliArgs = {
   dryRun?: boolean;
   network?: boolean;
   pauseOnError?: boolean;
+  quietExitCode?: boolean;
   reporter?: Reporters;
   wsEndpoint?: string;
   sandbox?: boolean;
diff --git a/src/parse_args.ts b/src/parse_args.ts
index 07824e9..36f578f 100644
--- a/src/parse_args.ts
+++ b/src/parse_args.ts
@@ -98,6 +98,10 @@ program
     '--pause-on-error',
     'pause on error until a keypress is made in the console. Useful during development'
   )
+  .option(
+    '--quiet-exit-code',
+    'always return 0 as an exit code status, regardless of test pass / fail. Only return > 0 exit codes on internal errors where the suite could not be run'
+  )
   .version(version)
   .description('Run synthetic tests');
 
@@ -112,6 +116,7 @@ if (options.richEvents) {
   options.reporter = options.reporter ?? 'json';
   options.ssblocks = true;
   options.network = true;
+  options.quietExitCode = true;
 }
 
 if (options.capability) {


Test patch:
diff --git a/__tests__/cli.test.ts b/__tests__/cli.test.ts
index 46dd941..d073686 100644
--- a/__tests__/cli.test.ts
+++ b/__tests__/cli.test.ts
@@ -41,6 +41,15 @@ describe('CLI', () => {
     expect(await cli.exitCode).toBe(1);
   });
 
+  it('runs the suites with --quiet-exit-code, always exiting with 0', async () => {
+    const cli = new CLIMock([
+      join(FIXTURES_DIR, 'error.journey.ts'),
+      '--quiet-exit-code',
+    ]);
+    await cli.waitFor('boom');
+    expect(await cli.exitCode).toBe(0);
+  });
+
   it('produce json output  --json and reporter=json flag', async () => {
     const output = async args => {
       const cli = new CLIMock([join(FIXTURES_DIR, 'fake.journey.ts'), ...args]);
@@ -77,6 +86,7 @@ describe('CLI', () => {
   it('mimick new heartbeat with `--rich-events` flag', async () => {
     const cli = new CLIMock([
       join(FIXTURES_DIR, 'fake.journey.ts'),
+      join(FIXTURES_DIR, 'error.journey.ts'),
       '--rich-events',
     ]);
     await cli.waitFor('journey/end');
@@ -94,7 +104,7 @@ describe('CLI', () => {
     });
 
     expect(await cli.exitCode).toBe(0);
-  });
+  }, 30000);
 
   it('override screenshots with `--rich-events` flag', async () => {
     const cli = new CLIMock([
diff --git a/__tests__/index.test.ts b/__tests__/index.test.ts
index 140e959..8fdbed2 100644
--- a/__tests__/index.test.ts
+++ b/__tests__/index.test.ts
@@ -72,6 +72,7 @@ describe('Run', () => {
       match: 'check*',
       network: true,
       pauseOnError: true,
+      quietExitCode: true,
       reporter: 'json',
     };
     await run(options);