Repository: elastic/synthetics
Pull request #441 (2022-01-11 18:11:20)

Stop runner from hanging indefinitely within ubuntu docker images [elastic/beats#29681]
> ⚠️ There is more E2E test information [here](https://github.com/elastic/beats/issues/29495#issuecomment-1011068200) in case you'd like to use the Heartbeat image built from source.

# Summary

This PR contributes to elastic/beats#29681 as it makes the `synthetics` runner work with the `ubuntu` images we will use.

After this PR, the Heartbeat container should be able to execute the `synthetics` runner.


# The problem in detail

The Heartbeat container wasn't able to run `synthetics` to completion, as the runner would hang indefinitely.

It couldn't do so because it didn't have the necessary graphics library ([`mesagl`](https://www.mesa3d.org/)) available to make hardware acceleration work ([more specifically, it seems Chromium uses `osmesa` for off-screen rendering](https://docs.mesa3d.org/osmesa.html)) and `newPage` not to hang if called too soon.

> ⚠️ The real problem here btw is not the lack of GL it seems, it's that the default GL used causes the `newPage` method to hang if we call it immediately after launching the browser. If we used `mesa` it seems that method would not hang, but I still have to manually test it.

**When that library wasn't available, and hardware acceleration was turned on, the creation of a new page would hang indefinitely if we _immediately_ tried to create a new page**. Just adding an `await new Promise(resolve => setTimeout(resolve, 5000))` statement after the browser launch did make journeys work given it caused `newPage` to return (instead of hanging).

**Therefore, I have disabled hardware acceleration by launching Chromium with the `--disable-gpu` flag, which makes it work fine in our Ubuntu image.** I decided to use that flag rather than changing the base image because [that's the exact fix that was applied in Playwright's master branch](https://github.com/microsoft/playwright/issues/4761).

> ⚠️ [You can see in this chromium mailing list thread that needing that flag to run within Linux containers is necessary](https://bugs.chromium.org/p/chromium/issues/detail?id=737678).


# How to test this PR

1. Build the Heartbeat image (your `cwd` must be the `x-pack/heartbeat` folder) using `env PLATFORMS="+all linux/amd64" mage package`
2. Run the image overwriting the `entrypoint` with a script similar to the one below
    ```
    #!/bin/bash
    docker run \
      -it \
      --entrypoint=/bin/bash \
      --rm \
      --name=heartbeat \
      --user=heartbeat \
      --volume="$PWD/heartbeat.yml:/usr/share/heartbeat/heartbeat.yml:ro" \
      --volume="$PWD/monitors.d:/usr/share/heartbeat/monitors.d:ro" \
      --volume="/your_path_to_the_synthetics_module/synthetics:/usr/share/synthetics" \
      docker.elastic.co/beats/heartbeat:8.1.0
    ```
    This will execute the image you've just built.
3. Once inside the container, you can either build the mounted `synthetics` module and run `dist/cli.js` or `npm link` it, and then run `heartbeat`, which will cause it to use the locally linked module.

The process is the same for `elastic-agent`, just make sure you change the image name to `elastic-agent-complete`.


# How I debugged it (in case it's helpful for others in a next opportunity):

* When executing the runner within the container, do so through building a mounted directory and running `dist/cli.js` with `node` (whenever you change a file, make sure to `npm run build`)
* Expose the container's `9229` port through the docker arg `-p 9229:9229` so that you can connect to the Node debugger in `chrome://inspect` from `localhost` when running `node --inspect-brk=0.0.0.0 dist/cli.js my_synthetic_path` (this is how I figured out that just waiting would not make the process hang, because as I stepped through lines that was enough time for the browser to init itself)

apmmachine: 
## :broken_heart: Build Failed


<!-- BUILD BADGES-->
> _the below badges are clickable and redirect to their specific view in the CI or DOCS_
[![Pipeline View](https://img.shields.io/badge/pipeline-pipeline%20-green)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-rum%2Felastic-synthetics%2FPR-441/detail/PR-441/1//pipeline) [![Test View](https://img.shields.io/badge/test-test-green)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-rum%2Felastic-synthetics%2FPR-441/detail/PR-441/1//tests) [![Changes](https://img.shields.io/badge/changes-changes-green)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-rum%2Felastic-synthetics%2FPR-441/detail/PR-441/1//changes) [![Artifacts](https://img.shields.io/badge/artifacts-artifacts-yellow)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-rum%2Felastic-synthetics%2FPR-441/detail/PR-441/1//artifacts) [![preview](https://img.shields.io/badge/docs-preview-yellowgreen)](http://synthetics_441.docs-preview.app.elstc.co/diff)  [![preview](https://img.shields.io/badge/elastic-observability-blue)](https://ci-stats.elastic.co/app/apm/services/apm-ci/transactions/view?rangeFrom=2022-01-11T18:01:36.326Z&rangeTo=2022-01-11T18:21:36.326Z&transactionName=apm-agent-rum%2Felastic-synthetics%2FPR-%7Bnumber%7D&transactionType=unknown&latencyAggregationType=avg&traceId=c0f5f1c99716d87c6df4f8089ea835c3&transactionId=1c62b7403e786450)

<!-- BUILD SUMMARY-->
<details><summary>Expand to view the summary</summary>
<p>

#### Build stats




* Start Time: 2022-01-11T18:11:36.326+0000


* Duration: 7 min 35 sec


* Commit: aafdb493b84b072e459abd1eceb3bcf8e64a69bb




</p>
</details>

<!-- TEST RESULTS IF ANY-->


<!-- STEPS ERRORS IF ANY -->











### Steps errors [![2](https://img.shields.io/badge/2%20-red)](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-rum%2Felastic-synthetics%2FPR-441/detail/PR-441/1//pipeline)
  <details><summary>Expand to view the steps failures</summary>
  <p>


  
  
  
  
##### `Install precommit hooks`
<ul>
<li>Took 0 min 1 sec . View more details <a href="https://apm-ci.elastic.co//blue/rest/organizations/jenkins/pipelines/apm-agent-rum/pipelines/elastic-synthetics/pipelines/PR-441/runs/1/steps/229/log/?start=0">here</a></li>
<li>Description: <code>          export PATH=/var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-441/bin:/var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-441/.nvm/versions/node/v14.17.5/bin:/command:/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin          ## Install with the hooks therefore ~/.cache/pre-commit will be created with the repos          pre-commit install --install-hooks        </code></l1>
</ul>
  
  
  
  
##### `Install precommit hooks`
<ul>
<li>Took 0 min 1 sec . View more details <a href="https://apm-ci.elastic.co//blue/rest/organizations/jenkins/pipelines/apm-agent-rum/pipelines/elastic-synthetics/pipelines/PR-441/runs/1/steps/233/log/?start=0">here</a></li>
<li>Description: <code>          export PATH=/var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-441/bin:/var/lib/jenkins/workspace/nt-rum_elastic-synthetics_PR-441/.nvm/versions/node/v14.17.5/bin:/command:/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11R6/bin          ## Install with the hooks therefore ~/.cache/pre-commit will be created with the repos          pre-commit install --install-hooks        </code></l1>
</ul>
  
  </p>
  </details>


## :robot: GitHub comments

To re-run your PR in the CI, just comment with:



- `/test` : Re-trigger the build.




<!--COMMENT_GENERATED_WITH_ID_comment.id-->
lucasfcosta: @vigneshshanmugam I have now:

* Tested this PR with the 7.16.3 images and they work fine.
    * for the old Heartbeat image
    * for the old Elastic Agent Image
* I have now updated this PR so that we only enable the `disable-gpu` for `headless` runs
* I have gathered trace metrics by:
    1. Running a journey which visits www.blueyard.com (a site which uses WebGL heavily) using the `--rich-events` flag
    2. Piping those results into `jq` and filtering the metric outputs
    The resulting command was:
    `synthetics --rich-events ../synth-example | jq -c 'select(.type == "step/metrics")' > /tmp/with-gpu-enabled.json`

Here are the results I've obtained for GPU enabled and disabled settings:

**with-gpu-enabled.json**
```
{"type":"step/metrics","@timestamp":1642072821592514.5,"journey":{"name":"my cool journey","id":"my cool journey"},"step":{"name":"google","index":1},"root_fields":{"browser":{"relative_trace":{"name":"navigationStart","type":"mark","start":{"us":10673903804}}},"os":{"platform":"darwin"},"package":{"name":"@elastic/synthetics","version":"1.0.0-beta.18"}},"package_version":"1.0.0-beta.18"}
{"type":"step/metrics","@timestamp":1642072821592701,"journey":{"name":"my cool journey","id":"my cool journey"},"step":{"name":"google","index":1},"root_fields":{"browser":{"relative_trace":{"name":"firstContentfulPaint","type":"mark","start":{"us":10674706722}}},"os":{"platform":"darwin"},"package":{"name":"@elastic/synthetics","version":"1.0.0-beta.18"}},"package_version":"1.0.0-beta.18"}
{"type":"step/metrics","@timestamp":1642072821592801.8,"journey":{"name":"my cool journey","id":"my cool journey"},"step":{"name":"google","index":1},"root_fields":{"browser":{"relative_trace":{"name":"largestContentfulPaint","type":"mark","start":{"us":10675141846}}},"os":{"platform":"darwin"},"package":{"name":"@elastic/synthetics","version":"1.0.0-beta.18"}},"package_version":"1.0.0-beta.18"}
{"type":"step/metrics","@timestamp":1642072821592873.5,"journey":{"name":"my cool journey","id":"my cool journey"},"step":{"name":"google","index":1},"root_fields":{"browser":{"relative_trace":{"name":"domContentLoaded","type":"mark","start":{"us":10675776934}}},"os":{"platform":"darwin"},"package":{"name":"@elastic/synthetics","version":"1.0.0-beta.18"}},"package_version":"1.0.0-beta.18"}
{"type":"step/metrics","@timestamp":1642072821593010.5,"journey":{"name":"my cool journey","id":"my cool journey"},"step":{"name":"google","index":1},"root_fields":{"browser":{"relative_trace":{"name":"loadEvent","type":"mark","start":{"us":10685391016}}},"os":{"platform":"darwin"},"package":{"name":"@elastic/synthetics","version":"1.0.0-beta.18"}},"package_version":"1.0.0-beta.18"}
{"type":"step/metrics","@timestamp":1642072821593085,"journey":{"name":"my cool journey","id":"my cool journey"},"step":{"name":"google","index":1},"root_fields":{"browser":{"relative_trace":{"name":"layoutShift","type":"mark","start":{"us":10677093709},"score":2.2897720336914065e-06}},"os":{"platform":"darwin"},"package":{"name":"@elastic/synthetics","version":"1.0.0-beta.18"}},"package_version":"1.0.0-beta.18"}
{"type":"step/metrics","@timestamp":1642072821593172.5,"journey":{"name":"my cool journey","id":"my cool journey"},"step":{"name":"google","index":1},"root_fields":{"browser":{"experience":{"cls":2.2897720336914065e-06,"fcp":{"us":802918},"lcp":{"us":1238042},"dcl":{"us":1873130},"load":{"us":11487212}}},"os":{"platform":"darwin"},"package":{"name":"@elastic/synthetics","version":"1.0.0-beta.18"}},"package_version":"1.0.0-beta.18"}
```

**with-gpu-disabled.json**
```
{"type":"step/metrics","@timestamp":1642072775738702.2,"journey":{"name":"my cool journey","id":"my cool journey"},"step":{"name":"google","index":1},"root_fields":{"browser":{"relative_trace":{"name":"navigationStart","type":"mark","start":{"us":10628714031}}},"os":{"platform":"darwin"},"package":{"name":"@elastic/synthetics","version":"1.0.0-beta.18"}},"package_version":"1.0.0-beta.18"}
{"type":"step/metrics","@timestamp":1642072775738786.8,"journey":{"name":"my cool journey","id":"my cool journey"},"step":{"name":"google","index":1},"root_fields":{"browser":{"relative_trace":{"name":"firstContentfulPaint","type":"mark","start":{"us":10629456146}}},"os":{"platform":"darwin"},"package":{"name":"@elastic/synthetics","version":"1.0.0-beta.18"}},"package_version":"1.0.0-beta.18"}
{"type":"step/metrics","@timestamp":1642072775738829.2,"journey":{"name":"my cool journey","id":"my cool journey"},"step":{"name":"google","index":1},"root_fields":{"browser":{"relative_trace":{"name":"largestContentfulPaint","type":"mark","start":{"us":10629827301}}},"os":{"platform":"darwin"},"package":{"name":"@elastic/synthetics","version":"1.0.0-beta.18"}},"package_version":"1.0.0-beta.18"}
{"type":"step/metrics","@timestamp":1642072775738869.8,"journey":{"name":"my cool journey","id":"my cool journey"},"step":{"name":"google","index":1},"root_fields":{"browser":{"relative_trace":{"name":"domContentLoaded","type":"mark","start":{"us":10630517348}}},"os":{"platform":"darwin"},"package":{"name":"@elastic/synthetics","version":"1.0.0-beta.18"}},"package_version":"1.0.0-beta.18"}
{"type":"step/metrics","@timestamp":1642072775738899.2,"journey":{"name":"my cool journey","id":"my cool journey"},"step":{"name":"google","index":1},"root_fields":{"browser":{"relative_trace":{"name":"loadEvent","type":"mark","start":{"us":10639842605}}},"os":{"platform":"darwin"},"package":{"name":"@elastic/synthetics","version":"1.0.0-beta.18"}},"package_version":"1.0.0-beta.18"}
{"type":"step/metrics","@timestamp":1642072775738926,"journey":{"name":"my cool journey","id":"my cool journey"},"step":{"name":"google","index":1},"root_fields":{"browser":{"relative_trace":{"name":"layoutShift","type":"mark","start":{"us":10631345774},"score":2.2009920535816084e-06}},"os":{"platform":"darwin"},"package":{"name":"@elastic/synthetics","version":"1.0.0-beta.18"}},"package_version":"1.0.0-beta.18"}
{"type":"step/metrics","@timestamp":1642072775738955.5,"journey":{"name":"my cool journey","id":"my cool journey"},"step":{"name":"google","index":1},"root_fields":{"browser":{"experience":{"cls":2.2009920535816084e-06,"fcp":{"us":742115},"lcp":{"us":1113270},"dcl":{"us":1803317},"load":{"us":11128574}}},"os":{"platform":"darwin"},"package":{"name":"@elastic/synthetics","version":"1.0.0-beta.18"}},"package_version":"1.0.0-beta.18"}
```

The experience summary in a table:

|      | With GPU Enabled         | With GPU Disabled        |
|------|--------------------------|--------------------------|
| CLS  | 0.0000022897720336914065 | 0.0000022009920535816084 |
| FCP (micros) | 802918                   | 742115                   |
| DCL (micros) | 1873130                  | 1803317                  |
| LOAD (micros) | 11487212                 | 11128574                 |
lucasfcosta: As mentioned by @vigneshshanmugam we'll drop further analysis on these given there's not much variation between runs and that they don't make much sense in headless mode:

> @lucasfcosta Lets drop the perf measurement, I just realized they dont make much sense in headless mode. Lots of optmisations are dropped anyways. Apologies for not thinking through much here.

Will merge as soon as CI passes and release.

Patch:
diff --git a/src/core/gatherer.ts b/src/core/gatherer.ts
index b73c9ed..86f369b 100644
--- a/src/core/gatherer.ts
+++ b/src/core/gatherer.ts
@@ -45,7 +45,13 @@ export class Gatherer {
         log(`Gatherer: connecting to WS endpoint: ${wsEndpoint}`);
         Gatherer.browser = await chromium.connect({ wsEndpoint });
       } else {
-        Gatherer.browser = await chromium.launch(playwrightOptions);
+        Gatherer.browser = await chromium.launch({
+          ...playwrightOptions,
+          args: [
+            ...(playwrightOptions?.headless ? ['--disable-gpu'] : []),
+            ...(playwrightOptions?.args ?? []),
+          ],
+        });
       }
     }
     const context = await Gatherer.browser.newContext({


Test patch:
diff --git a/__tests__/core/gatherer.test.ts b/__tests__/core/gatherer.test.ts
index d3f3e8c..3b8740a 100644
--- a/__tests__/core/gatherer.test.ts
+++ b/__tests__/core/gatherer.test.ts
@@ -29,6 +29,7 @@ import { wsEndpoint } from '../utils/test-config';
 import { devices } from 'playwright-chromium';
 import { Server } from '../utils/server';
 import { megabitsToBytes } from '../../src/helpers';
+import { chromium } from 'playwright-chromium';
 
 jest.mock('../../src/plugins/network');
 
@@ -42,12 +43,60 @@ describe('Gatherer', () => {
     await server.close();
   });
 
+  afterEach(() => {
+    jest.restoreAllMocks();
+  });
+
   it('boot and close browser', async () => {
     const driver = await Gatherer.setupDriver({ wsEndpoint });
     expect(typeof driver.page.goto).toBe('function');
     await Gatherer.stop();
   });
 
+  // This test should only run when a browser service is up
+  (wsEndpoint ? it : it.skip)(
+    'does not the disable-gpu flag to start browser when running headful',
+    async () => {
+      const chromiumLaunch = jest
+        .spyOn(chromium, 'launch')
+        .mockImplementation(() => {
+          return chromium.connect({ wsEndpoint });
+        });
+
+      await Gatherer.setupDriver({
+        playwrightOptions: { headless: false },
+      });
+      expect(chromiumLaunch).toHaveBeenCalledWith(
+        expect.objectContaining({
+          args: expect.not.arrayContaining(['--disable-gpu']),
+        })
+      );
+      await Gatherer.stop();
+    }
+  );
+
+  // This test should only run when a browser service is up
+  (wsEndpoint ? it : it.skip)(
+    'uses the disable-gpu flag to start browser when running headlessly',
+    async () => {
+      const chromiumLaunch = jest
+        .spyOn(chromium, 'launch')
+        .mockImplementation(() => {
+          return chromium.connect({ wsEndpoint });
+        });
+
+      await Gatherer.setupDriver({
+        playwrightOptions: { headless: true },
+      });
+      expect(chromiumLaunch).toHaveBeenCalledWith(
+        expect.objectContaining({
+          args: expect.arrayContaining(['--disable-gpu']),
+        })
+      );
+      await Gatherer.stop();
+    }
+  );
+
   it('setup and dispose driver', async () => {
     const driver = await Gatherer.setupDriver({ wsEndpoint });
     await Gatherer.dispose(driver);