Repository: GoogleChrome/lighthouse
Pull request #9101 (2019-06-01 01:13:58)

core: gather source maps
#9097

Gather source maps.

Nothing done with them yet.



Hoten: WIP b/c no tests, and:

I want to put source maps in a new artifact, because gathering them is a lot of network requests and parsing that should only be done if an audit explicitly needs it - which, right now, does not exist.

It could be in a new gatherer, but I'd like to use the Debugger.onScriptParsed to collect the `sourceMapURL` (instead of parsing it ourselves). So we can add `sourceMapURL` to `ScriptElements`, and make a computed artifact, cool. But we don't have the driver in computed artifacts (afaict), so can't make fetch requests in the page to collect the source maps.

Is there a nice way to pass the driver to a computed artifact?
patrickhulce: > Is there a nice way to pass the driver to a computed artifact?

Sadly, no. Computed artifacts are firmly in audit phase territory and there's no way to have dependencies between artifacts unless one is a `BaseArtifact`

> Or is it better to forgo Debugger.onScriptParsed and just parse for sourceMapURL?

I'm not sure I understand the decision here, since it sounds like if you needed a separate gatherer either way you could have the separate one listen on `Debugger.onScriptParsed` for now? At some point in the future it might make sense to merge these two if the set of audits that need them overlap significantly enough.
Hoten: > I'm not sure I understand the decision here, since it sounds like if you needed a separate gatherer either way you could have the separate one listen on Debugger.onScriptParsed for now?

Failed to realize it's not required to do that in the `script-elements`. yeah separate gatherer makes 100% sense now.
codecov[bot]: # [Codecov](https://codecov.io/gh/GoogleChrome/lighthouse/pull/9101?src=pr&el=h1) Report
> Merging [#9101](https://codecov.io/gh/GoogleChrome/lighthouse/pull/9101?src=pr&el=desc) into [master](https://codecov.io/gh/GoogleChrome/lighthouse/commit/0eacf068b43f9306e9055c1d0ff19987d4fde56c?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `94.87%`.

[![Impacted file tree graph](https://codecov.io/gh/GoogleChrome/lighthouse/pull/9101/graphs/tree.svg?width=650&token=rZ1iVZWtql&height=150&src=pr)](https://codecov.io/gh/GoogleChrome/lighthouse/pull/9101?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #9101      +/-   ##
==========================================
- Coverage   91.44%   91.43%   -0.02%     
==========================================
  Files         291      292       +1     
  Lines        9929     9981      +52     
==========================================
+ Hits         9080     9126      +46     
- Misses        849      855       +6
```

| Flag | Coverage Δ | |
|---|---|---|
| #smoke | `84.43% <94.87%> (-0.33%)` | :arrow_down: |
| #unit | `89.24% <15.38%> (-0.29%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/GoogleChrome/lighthouse/pull/9101?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [lighthouse-core/config/default-config.js](https://codecov.io/gh/GoogleChrome/lighthouse/pull/9101/diff?src=pr&el=tree#diff-bGlnaHRob3VzZS1jb3JlL2NvbmZpZy9kZWZhdWx0LWNvbmZpZy5qcw==) | `87.5% <ø> (ø)` | :arrow_up: |
| [lighthouse-core/gather/gatherers/source-maps.js](https://codecov.io/gh/GoogleChrome/lighthouse/pull/9101/diff?src=pr&el=tree#diff-bGlnaHRob3VzZS1jb3JlL2dhdGhlci9nYXRoZXJlcnMvc291cmNlLW1hcHMuanM=) | `94.87% <94.87%> (ø)` | |
| [lighthouse-core/audits/user-timings.js](https://codecov.io/gh/GoogleChrome/lighthouse/pull/9101/diff?src=pr&el=tree#diff-bGlnaHRob3VzZS1jb3JlL2F1ZGl0cy91c2VyLXRpbWluZ3MuanM=) | `96% <0%> (-4%)` | :arrow_down: |
| [...house-core/computed/metrics/lantern-speed-index.js](https://codecov.io/gh/GoogleChrome/lighthouse/pull/9101/diff?src=pr&el=tree#diff-bGlnaHRob3VzZS1jb3JlL2NvbXB1dGVkL21ldHJpY3MvbGFudGVybi1zcGVlZC1pbmRleC5qcw==) | `97.14% <0%> (-2.86%)` | :arrow_down: |
| [lighthouse-core/gather/gather-runner.js](https://codecov.io/gh/GoogleChrome/lighthouse/pull/9101/diff?src=pr&el=tree#diff-bGlnaHRob3VzZS1jb3JlL2dhdGhlci9nYXRoZXItcnVubmVyLmpz) | `99.05% <0%> (-0.95%)` | :arrow_down: |
| [lighthouse-core/lib/manifest-parser.js](https://codecov.io/gh/GoogleChrome/lighthouse/pull/9101/diff?src=pr&el=tree#diff-bGlnaHRob3VzZS1jb3JlL2xpYi9tYW5pZmVzdC1wYXJzZXIuanM=) | `90.55% <0%> (+1.07%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/GoogleChrome/lighthouse/pull/9101?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/GoogleChrome/lighthouse/pull/9101?src=pr&el=footer). Last update [0eacf06...5da78cf](https://codecov.io/gh/GoogleChrome/lighthouse/pull/9101?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Hoten: Should we do some explicit property checks for the provided map? Right now it's assumed valid for any valid JSON.
Hoten: @paulirish 	@brendankenny 	thoughts?
connorjclark: Added unit tests (and even caught a bug :) )

remove smokes?
connorjclark: bump @brendankenny 	@patrickhulce 	
patrickhulce: Travis is complaining about MetaElements in the config though?
connorjclark: > Travis is complaining about MetaElements in the config though?

fixed
googlebot: We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors.  If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA ([login here](https://cla.developers.google.com/) to double check)?  If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2FGoogleChrome%2Flighthouse%2Fpull%2F9101) for more info**.

<!-- need_author_cla -->
paulirish: very excited to have this!

Tried this out but encountered some problems.

My repro:

1. apply this diff: 
```diff
diff --git a/lighthouse-core/audits/network-requests.js b/lighthouse-core/audits/network-requests.js
index 53acc72b..a58d9f7a 100644
--- a/lighthouse-core/audits/network-requests.js
+++ b/lighthouse-core/audits/network-requests.js
@@ -19,7 +19,7 @@ class NetworkRequests extends Audit {
       scoreDisplayMode: Audit.SCORING_MODES.INFORMATIVE,
       title: 'Network Requests',
       description: 'Lists the network requests that were made during page load.',
-      requiredArtifacts: ['devtoolsLogs'],
+      requiredArtifacts: ['devtoolsLogs', 'SourceMaps'],
     };
   }
 
diff --git a/lighthouse-core/config/default-config.js b/lighthouse-core/config/default-config.js
index d10e9119..d7edd02b 100644
--- a/lighthouse-core/config/default-config.js
+++ b/lighthouse-core/config/default-config.js
@@ -124,6 +124,7 @@ const defaultConfig = {
       'link-elements',
       'meta-elements',
       'script-elements',
+      'source-maps',
       'dobetterweb/appcache',
       'dobetterweb/doctype',
       'dobetterweb/domstats',
```

2. run `lighthouse https://www.nytimes.com/section/science -G`
3. run `node -e "console.log(require('./latest-run/artifacts.json').SourceMaps)"`

I see a lot of protocol timeout, mostly: 
![image](https://user-images.githubusercontent.com/39191/61009737-937d6a00-a328-11e9-9ee3-d702bcba37ef.png)

connorjclark: re: timeouts - I guess 250 is much too low. https://github.com/GoogleChrome/lighthouse/pull/9101/files/a5f035e8bcb244816a25d34c406208b4659ec6b1#diff-7519d8c2231119d4a3feee7c6c8e8cd5R48

```
cjamcl@cjamcl:~/code/chromium/src$ time curl -I https://news.google.com/swg/js/v1/swg.js.map
HTTP/2 200 
accept-ranges: bytes
vary: Accept-Encoding
content-type: text/html
content-length: 243582
date: Wed, 10 Jul 2019 22:54:06 GMT
expires: Wed, 10 Jul 2019 23:44:06 GMT
cache-control: public, max-age=3000
last-modified: Mon, 01 Jul 2019 17:18:51 GMT
x-content-type-options: nosniff
server: sffe
x-xss-protection: 0
alt-svc: quic=":443"; ma=2592000; v="46,43,39"


real	0m0.265s
user	0m0.013s
sys	0m0.009s

```
connorjclark: also: should check `x-sourcemap` header for URL
connorjclark: updated to resolve to base url.

note that you'll often see `SyntaxError: Unexpected token < in JSON at position 0` if the source map is behind an authenticated check. example: https://platform.iteratehq.com/sdk-prod-f5756fbc6a6363b60247.js

i'll add some checking to the fetch response status code so we provide a better error message.
connorjclark: Thoughts on performance now that requests are parallelized?

@patrickhulce specific changes you'd like to see?
patrickhulce: Oh no specific requests @connorjclark, I was just saying I approved with the thought that it was a fine stepping stone in the larger bundle analysis project that would see lots of changes before actually being surfaced anywhere, but if the goal is to make this ready to be consumed the approval stamp on a shifted goal didn't feel quite right.

Happy to give this another look with fresh eyes and that mindset if you'd like/we don't already have too many cooks :)
connorjclark: >  I was just saying I approved with the thought that it was a fine stepping stone in the larger bundle analysis project that would see lots of changes before actually being surfaced anywhere

FWIW this is still my intention. It's not in the default config, and no audits use it yet, so this is truly a first pass liable to change when audits using it show any problems.
paulirish: thx very much for all the back n forth! 
googlebot: A Googler has manually verified that the CLAs look good.

(Googler, please make sure the reason for overriding the CLA status is clearly documented in these comments.)

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2FGoogleChrome%2Flighthouse%2Fpull%2F9101) for more info**.

<!-- cla_yes -->
googlebot: We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors.  If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA ([login here](https://cla.developers.google.com/) to double check)?  If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2FGoogleChrome%2Flighthouse%2Fpull%2F9101) for more info**.

<!-- need_author_cla -->
googlebot: A Googler has manually verified that the CLAs look good.

(Googler, please make sure the reason for overriding the CLA status is clearly documented in these comments.)

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2FGoogleChrome%2Flighthouse%2Fpull%2F9101) for more info**.

<!-- cla_yes -->
googlebot: We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors.  If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA ([login here](https://cla.developers.google.com/) to double check)?  If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2FGoogleChrome%2Flighthouse%2Fpull%2F9101) for more info**.

<!-- need_author_cla -->
googlebot: A Googler has manually verified that the CLAs look good.

(Googler, please make sure the reason for overriding the CLA status is clearly documented in these comments.)

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2FGoogleChrome%2Flighthouse%2Fpull%2F9101) for more info**.

<!-- cla_yes -->
connorjclark: @brendankenny you're marked "changes requested", is that still accurate
connorjclark: CORs is preventing fetching maps. Ex:

`https://www.nytimes.com/section/science`
```
{
      "scriptUrl": "https://news.google.com/swg/js/v1/swg.js",
      "sourceMapUrl": "https://news.google.com/swg/js/v1/swg.js.map",
      "errorMessage": "TypeError: Failed to fetch"
    }
```

fetching `sourceMapUrl` on nyt:
![image](https://user-images.githubusercontent.com/4071474/61682931-76ea1600-acc8-11e9-946a-56ed2e954959.png)

Not sure how to get around this. Our only access to the network AFAIK is via the page's fetch over the protocol. Using any node api is not allowed since we have bundling constraints.
googlebot: We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors.  If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA ([login here](https://cla.developers.google.com/) to double check)?  If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2FGoogleChrome%2Flighthouse%2Fpull%2F9101) for more info**.

<!-- need_author_cla -->
googlebot: A Googler has manually verified that the CLAs look good.

(Googler, please make sure the reason for overriding the CLA status is clearly documented in these comments.)

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2FGoogleChrome%2Flighthouse%2Fpull%2F9101) for more info**.

<!-- cla_yes -->
connorjclark: Considerable work needs to be done to support fetching arbitrary resources outside the page context for each of our environments. We have:

1) node
2) devtools frontend
3) lightrider

1) is easy - the `http` module.
2) is also easy - DevTools has a host API for this.
3) is not so easy.

Options moving forward:

a) land as-is, some maps can't be fetched due to CORs. For the audit that will come after, we can ignore fetch errors.
b) hold off landing completely until this is well supported.

I vote for a) to keep the PR smaller.
googlebot: We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors.  If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA ([login here](https://cla.developers.google.com/) to double check)?  If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2FGoogleChrome%2Flighthouse%2Fpull%2F9101) for more info**.

<!-- need_author_cla -->
connorjclark: alternatively, could do a hybrid: support node fully using `http` module and fallback to in-page fetch in other environments (or just disable).
brendankenny: > a) land as-is, some maps can't be fetched due to CORs. For the audit that will come after, we can ignore fetch errors.
> b) hold off landing completely until this is well supported.
> 
> I vote for a) to keep the PR smaller.

I definitely don't think we should block on that, at worst it's only as bad as if they hadn't published their source maps
googlebot: A Googler has manually verified that the CLAs look good.

(Googler, please make sure the reason for overriding the CLA status is clearly documented in these comments.)

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2FGoogleChrome%2Flighthouse%2Fpull%2F9101) for more info**.

<!-- cla_yes -->
googlebot: We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors.  If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA ([login here](https://cla.developers.google.com/) to double check)?  If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2FGoogleChrome%2Flighthouse%2Fpull%2F9101) for more info**.

<!-- need_author_cla -->
googlebot: A Googler has manually verified that the CLAs look good.

(Googler, please make sure the reason for overriding the CLA status is clearly documented in these comments.)

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2FGoogleChrome%2Flighthouse%2Fpull%2F9101) for more info**.

<!-- cla_yes -->
googlebot: We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors.  If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA ([login here](https://cla.developers.google.com/) to double check)?  If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2FGoogleChrome%2Flighthouse%2Fpull%2F9101) for more info**.

<!-- need_author_cla -->
connorjclark: it seems half of these comments could be @googlebot 👎 	
googlebot: A Googler has manually verified that the CLAs look good.

(Googler, please make sure the reason for overriding the CLA status is clearly documented in these comments.)

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2FGoogleChrome%2Flighthouse%2Fpull%2F9101) for more info**.

<!-- cla_yes -->
googlebot: We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors.  If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA ([login here](https://cla.developers.google.com/) to double check)?  If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2FGoogleChrome%2Flighthouse%2Fpull%2F9101) for more info**.

<!-- need_author_cla -->
connorjclark: @brendankenny 	@exterkamp 	ya'll wanna add your personal emails to the CLA thing? click the Googler link above

Patch:
diff --git a/lighthouse-core/gather/driver.js b/lighthouse-core/gather/driver.js
index b9d58f6d5..1d0cfea5d 100644
--- a/lighthouse-core/gather/driver.js
+++ b/lighthouse-core/gather/driver.js
@@ -476,7 +476,7 @@ class Driver {
     const contextId = options.useIsolation ? await this._getOrCreateIsolatedContextId() : undefined;
 
     try {
-      // `await` is not redunant here because we want to `catch` the async errors
+      // `await` is not redundant here because we want to `catch` the async errors
       return await this._evaluateInContext(expression, contextId);
     } catch (err) {
       // If we were using isolation and the context disappeared on us, retry one more time.
diff --git a/lighthouse-core/gather/gatherers/source-maps.js b/lighthouse-core/gather/gatherers/source-maps.js
new file mode 100644
index 000000000..17f7405af
--- /dev/null
+++ b/lighthouse-core/gather/gatherers/source-maps.js
@@ -0,0 +1,159 @@
+/**
+ * @license Copyright 2019 Google Inc. All Rights Reserved.
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
+ */
+'use strict';
+
+/** @typedef {import('../driver.js')} Driver */
+
+const Gatherer = require('./gatherer.js');
+const URL = require('../../lib/url-shim.js');
+
+/**
+ * This function fetches source maps; it is careful not to parse the response as JSON, as it will
+ * just need to be serialized again over the protocol, and source maps can
+ * be huge.
+ *
+ * @param {string} url
+ * @return {Promise<string>}
+ */
+/* istanbul ignore next */
+async function fetchSourceMap(url) {
+  // eslint-disable-next-line no-undef
+  const response = await fetch(url);
+  if (response.ok) {
+    return response.text();
+  } else {
+    throw new Error(`Received status code ${response.status} for ${url}`);
+  }
+}
+
+/**
+ * @fileoverview Gets JavaScript source maps.
+ */
+class SourceMaps extends Gatherer {
+  constructor() {
+    super();
+    /** @type {LH.Crdp.Debugger.ScriptParsedEvent[]} */
+    this._scriptParsedEvents = [];
+    this.onScriptParsed = this.onScriptParsed.bind(this);
+  }
+
+  /**
+   * @param {Driver} driver
+   * @param {string} sourceMapUrl
+   * @return {Promise<LH.Artifacts.RawSourceMap>}
+   */
+  async fetchSourceMapInPage(driver, sourceMapUrl) {
+    driver.setNextProtocolTimeout(1500);
+    /** @type {string} */
+    const sourceMapJson =
+      await driver.evaluateAsync(`(${fetchSourceMap})(${JSON.stringify(sourceMapUrl)})`);
+    return JSON.parse(sourceMapJson);
+  }
+
+  /**
+   * @param {string} sourceMapURL
+   * @return {LH.Artifacts.RawSourceMap}
+   */
+  parseSourceMapFromDataUrl(sourceMapURL) {
+    const buffer = Buffer.from(sourceMapURL.split(',')[1], 'base64');
+    return JSON.parse(buffer.toString());
+  }
+
+  /**
+   * @param {LH.Crdp.Debugger.ScriptParsedEvent} event
+   */
+  onScriptParsed(event) {
+    if (event.sourceMapURL) {
+      this._scriptParsedEvents.push(event);
+    }
+  }
+
+  /**
+   * @param {LH.Gatherer.PassContext} passContext
+   */
+  async beforePass(passContext) {
+    const driver = passContext.driver;
+    driver.on('Debugger.scriptParsed', this.onScriptParsed);
+    await driver.sendCommand('Debugger.enable');
+  }
+
+  /**
+   * @param {string} url
+   * @param {string} base
+   * @return {string|undefined}
+   */
+  _resolveUrl(url, base) {
+    try {
+      return new URL(url, base).href;
+    } catch (e) {
+      return;
+    }
+  }
+
+  /**
+   * @param {Driver} driver
+   * @param {LH.Crdp.Debugger.ScriptParsedEvent} event
+   * @return {Promise<LH.Artifacts.SourceMap>}
+   */
+  async _retrieveMapFromScriptParsedEvent(driver, event) {
+    if (!event.sourceMapURL) {
+      throw new Error('precondition failed: event.sourceMapURL should exist');
+    }
+
+    // `sourceMapURL` is simply the URL found in either a magic comment or an x-sourcemap header.
+    // It has not been resolved to a base url.
+    const isSourceMapADataUri = event.sourceMapURL.startsWith('data:');
+    const scriptUrl = event.url;
+    const rawSourceMapUrl = isSourceMapADataUri ?
+        event.sourceMapURL :
+        this._resolveUrl(event.sourceMapURL, event.url);
+
+    if (!rawSourceMapUrl) {
+      return {
+        scriptUrl,
+        errorMessage: `Could not resolve map url: ${event.sourceMapURL}`,
+      };
+    }
+
+    // sourceMapUrl isn't included in the the artifact if it was a data URL.
+    const sourceMapUrl = isSourceMapADataUri ? undefined : rawSourceMapUrl;
+
+    try {
+      const map = isSourceMapADataUri ?
+          this.parseSourceMapFromDataUrl(rawSourceMapUrl) :
+          await this.fetchSourceMapInPage(driver, rawSourceMapUrl);
+      return {
+        scriptUrl,
+        sourceMapUrl,
+        map,
+      };
+    } catch (err) {
+      return {
+        scriptUrl,
+        sourceMapUrl,
+        errorMessage: err.toString(),
+      };
+    }
+  }
+
+  /**
+   * @param {LH.Gatherer.PassContext} passContext
+   * @return {Promise<LH.Artifacts['SourceMaps']>}
+   */
+  async afterPass(passContext) {
+    const driver = passContext.driver;
+
+    driver.off('Debugger.scriptParsed', this.onScriptParsed);
+    await driver.sendCommand('Debugger.disable');
+
+    const eventProcessPromises = this._scriptParsedEvents
+      .map((event) => this._retrieveMapFromScriptParsedEvent(driver, event));
+
+    return Promise.all(eventProcessPromises);
+  }
+}
+
+module.exports = SourceMaps;
diff --git a/types/artifacts.d.ts b/types/artifacts.d.ts
index 8dda838f1..9e93814da 100644
--- a/types/artifacts.d.ts
+++ b/types/artifacts.d.ts
@@ -123,6 +123,8 @@ declare global {
       RobotsTxt: {status: number|null, content: string|null};
       /** Version information for all ServiceWorkers active after the first page load. */
       ServiceWorker: {versions: Crdp.ServiceWorker.ServiceWorkerVersion[], registrations: Crdp.ServiceWorker.ServiceWorkerRegistration[]};
+      /** Source maps of scripts executed in the page. */
+      SourceMaps: Array<Artifacts.SourceMap>;
       /** The status of an offline fetch of the page's start_url. -1 and a explanation if missing or there was an error. */
       StartUrl: {statusCode: number, explanation?: string};
       /** Information on <script> and <link> tags blocking first paint. */
@@ -215,6 +217,46 @@ declare global {
         requestId: string | null
       }
 
+      /** @see https://sourcemaps.info/spec.html#h.qz3o9nc69um5 */
+      export type RawSourceMap = {
+        /** File version and must be a positive integer. */
+        version: number
+        /** A list of original source files used by the `mappings` entry. */
+        sources: string[]
+        /** A list of symbol names used by the `mappings` entry. */
+        names?: string[]
+        /** An optional source root, useful for relocating source files on a server or removing repeated values in the `sources` entry. This value is prepended to the individual entries in the `source` field. */
+        sourceRoot?: string
+        /** An optional list of source content, useful when the `source` can’t be hosted. The contents are listed in the same order as the sources. */
+        sourcesContent?: string[]
+        /** A string with the encoded mapping data. */
+        mappings: string
+        /** An optional name of the generated code (the bundled code that was the result of this build process) that this source map is associated with. */
+        file?: string
+      }
+
+      /**
+       * Source map for a given script found at scriptUrl. If there is an error in fetching or
+       * parsing the map, errorMessage will be defined instead of map.
+       */
+      export type SourceMap = {
+        /** URL of code that source map applies to. */
+        scriptUrl: string
+        /** URL of the source map. undefined if from data URL. */
+        sourceMapUrl?: string
+        /** Source map data structure. */
+        map: RawSourceMap
+      } | {
+        /** URL of code that source map applies to. */
+        scriptUrl: string
+        /** URL of the source map. undefined if from data URL. */
+        sourceMapUrl?: string
+        /** Error that occurred during fetching or parsing of source map. */
+        errorMessage: string
+        /** No map on account of error. */
+        map?: undefined;
+      }
+
       /** @see https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a#Attributes */
       export interface AnchorElement {
         rel: string


Test patch:
diff --git a/lighthouse-core/test/gather/driver-test.js b/lighthouse-core/test/gather/driver-test.js
index 62bfbcb2c..d78ba2c97 100644
--- a/lighthouse-core/test/gather/driver-test.js
+++ b/lighthouse-core/test/gather/driver-test.js
@@ -10,6 +10,7 @@ const Connection = require('../../gather/connections/connection.js');
 const Element = require('../../lib/element.js');
 const EventEmitter = require('events').EventEmitter;
 const {protocolGetVersionResponse} = require('./fake-driver.js');
+const {createMockSendCommandFn, createMockOnceFn} = require('./mock-commands.js');
 
 const redirectDevtoolsLog = require('../fixtures/wikipedia-redirect.devtoolslog.json');
 
@@ -17,72 +18,6 @@ const redirectDevtoolsLog = require('../fixtures/wikipedia-redirect.devtoolslog.
 
 jest.useFakeTimers();
 
-/**
- * Creates a jest mock function whose implementation consumes mocked protocol responses matching the
- * requested command in the order they were mocked.
- *
- * It is decorated with two methods:
- *    - `mockResponse` which pushes protocol message responses for consumption
- *    - `findInvocation` which asserts that `sendCommand` was invoked with the given command and
- *      returns the protocol message argument.
- */
-function createMockSendCommandFn() {
-  const mockResponses = [];
-  const mockFn = jest.fn().mockImplementation(command => {
-    const indexOfResponse = mockResponses.findIndex(entry => entry.command === command);
-    if (indexOfResponse === -1) throw new Error(`${command} unimplemented`);
-    const {response, delay} = mockResponses[indexOfResponse];
-    mockResponses.splice(indexOfResponse, 1);
-    if (delay) return new Promise(resolve => setTimeout(() => resolve(response), delay));
-    return Promise.resolve(response);
-  });
-
-  mockFn.mockResponse = (command, response, delay) => {
-    mockResponses.push({command, response, delay});
-    return mockFn;
-  };
-
-  mockFn.findInvocation = command => {
-    expect(mockFn).toHaveBeenCalledWith(command, expect.anything());
-    return mockFn.mock.calls.find(call => call[0] === command)[1];
-  };
-
-  return mockFn;
-}
-
-/**
- * Creates a jest mock function whose implementation invokes `.on`/`.once` listeners after a setTimeout tick.
- * Closely mirrors `createMockSendCommandFn`.
- *
- * It is decorated with two methods:
- *    - `mockEvent` which pushes protocol event payload for consumption
- *    - `findListener` which asserts that `on` was invoked with the given event name and
- *      returns the listener .
- */
-function createMockOnceFn() {
-  const mockEvents = [];
-  const mockFn = jest.fn().mockImplementation((eventName, listener) => {
-    const indexOfResponse = mockEvents.findIndex(entry => entry.event === eventName);
-    if (indexOfResponse === -1) return;
-    const {response} = mockEvents[indexOfResponse];
-    mockEvents.splice(indexOfResponse, 1);
-    // Wait a tick because real events never fire immediately
-    setTimeout(() => listener(response), 0);
-  });
-
-  mockFn.mockEvent = (event, response) => {
-    mockEvents.push({event, response});
-    return mockFn;
-  };
-
-  mockFn.findListener = event => {
-    expect(mockFn).toHaveBeenCalledWith(event, expect.anything());
-    return mockFn.mock.calls.find(call => call[0] === event)[1];
-  };
-
-  return mockFn;
-}
-
 /**
  * Transparently augments the promise with inspectable functions to query its state.
  *
diff --git a/lighthouse-core/test/gather/gatherers/source-maps-test.js b/lighthouse-core/test/gather/gatherers/source-maps-test.js
new file mode 100644
index 000000000..5d809cbb7
--- /dev/null
+++ b/lighthouse-core/test/gather/gatherers/source-maps-test.js
@@ -0,0 +1,241 @@
+/**
+ * @license Copyright 2019 Google Inc. All Rights Reserved.
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
+ */
+'use strict';
+
+/* eslint-env jest */
+
+jest.useFakeTimers();
+
+const Driver = require('../../../gather/driver.js');
+const Connection = require('../../../gather/connections/connection.js');
+const SourceMaps = require('../../../gather/gatherers/source-maps.js');
+const {createMockSendCommandFn, createMockOnFn} = require('../mock-commands.js');
+
+const mapJson = JSON.stringify({
+  version: 3,
+  file: 'out.js',
+  sourceRoot: '',
+  sources: ['foo.js', 'bar.js'],
+  names: ['src', 'maps', 'are', 'fun'],
+  mappings: 'AAgBC,SAAQ,CAAEA',
+});
+
+describe('SourceMaps gatherer', () => {
+  /**
+   * `scriptParsedEvent` mocks the `sourceMapURL` and `url` seen from the protocol.
+   * `map` mocks the (JSON) of the source maps that `Runtime.evaluate` returns.
+   * `resolvedSourceMapUrl` is used to assert that the SourceMaps gatherer is using the expected
+   *                        url to fetch the source map.
+   * `fetchError` mocks an error that happens in the page. Only fetch error message make sense.
+   * @param {Array<{scriptParsedEvent: LH.Crdp.Debugger.ScriptParsedEvent, map: string, resolvedSourceMapUrl?: string, fetchError: string}>} mapsAndEvents
+   * @return {Promise<LH.Artifacts['SourceMaps']>}
+   */
+  async function runSourceMaps(mapsAndEvents) {
+    const onMock = createMockOnFn();
+
+    const sendCommandMock = createMockSendCommandFn()
+      .mockResponse('Debugger.enable', {})
+      .mockResponse('Debugger.disable', {});
+
+    for (const {scriptParsedEvent, map, resolvedSourceMapUrl, fetchError} of mapsAndEvents) {
+      onMock.mockEvent('protocolevent', {
+        method: 'Debugger.scriptParsed',
+        params: scriptParsedEvent,
+      });
+
+      if (scriptParsedEvent.sourceMapURL.startsWith('data:')) {
+        // Only the source maps that need to be fetched use the `evaluateAsync` code path.
+        continue;
+      }
+
+      if (map && fetchError) {
+        throw new Error('should only define map or fetchError, not both.');
+      }
+
+      sendCommandMock.mockResponse('Runtime.evaluate', ({expression}) => {
+        // Check that the source map url was resolved correctly. It'll be somewhere
+        // in the code sent to Runtime.evaluate.
+        if (resolvedSourceMapUrl && !expression.includes(resolvedSourceMapUrl)) {
+          throw new Error(`did not request expected url: ${resolvedSourceMapUrl}`);
+        }
+
+        const value = fetchError ?
+          Object.assign(new Error(), {message: fetchError, __failedInBrowser: true}) :
+          map;
+        return {result: {value}};
+      });
+    }
+    const connectionStub = new Connection();
+    connectionStub.sendCommand = sendCommandMock;
+    connectionStub.on = onMock;
+
+    const driver = new Driver(connectionStub);
+
+    const sourceMaps = new SourceMaps();
+    await sourceMaps.beforePass({driver});
+    jest.advanceTimersByTime(1);
+    return sourceMaps.afterPass({driver});
+  }
+
+  function makeJsonDataUrl(data) {
+    return 'data:application/json;charset=utf-8;base64,' + Buffer.from(data).toString('base64');
+  }
+
+  it('ignores script with no source map url', async () => {
+    const artifact = await runSourceMaps([
+      {
+        scriptParsedEvent: {
+          url: 'http://www.example.com/script.js',
+          sourceMapURL: '',
+        },
+        map: null,
+      },
+    ]);
+    expect(artifact).toEqual([]);
+  });
+
+  it('fetches map for script with source map url', async () => {
+    const mapsAndEvents = [
+      {
+        scriptParsedEvent: {
+          url: 'http://www.example.com/bundle.js',
+          sourceMapURL: 'http://www.example.com/bundle.js.map',
+        },
+        map: mapJson,
+        resolvedSourceMapUrl: 'http://www.example.com/bundle.js.map',
+      },
+    ];
+    const artifact = await runSourceMaps(mapsAndEvents);
+    expect(artifact).toEqual([
+      {
+        scriptUrl: mapsAndEvents[0].scriptParsedEvent.url,
+        sourceMapUrl: mapsAndEvents[0].scriptParsedEvent.sourceMapURL,
+        map: JSON.parse(mapsAndEvents[0].map),
+      },
+    ]);
+  });
+
+  it('fetches map for script with relative source map url', async () => {
+    const mapsAndEvents = [
+      {
+        scriptParsedEvent: {
+          url: 'http://www.example.com/path/bundle.js',
+          sourceMapURL: 'bundle.js.map',
+        },
+        map: mapJson,
+        resolvedSourceMapUrl: 'http://www.example.com/path/bundle.js.map',
+      },
+      {
+        scriptParsedEvent: {
+          url: 'http://www.example.com/path/bundle.js',
+          sourceMapURL: '../bundle.js.map',
+        },
+        map: mapJson,
+        resolvedSourceMapUrl: 'http://www.example.com/bundle.js.map',
+      },
+      {
+        scriptParsedEvent: {
+          url: 'http://www.example.com/path/bundle.js',
+          sourceMapURL: 'http://www.example-2.com/path/bundle.js',
+        },
+        map: mapJson,
+        resolvedSourceMapUrl: 'http://www.example-2.com/path/bundle.js',
+      },
+    ];
+    const artifacts = await runSourceMaps(mapsAndEvents);
+    expect(artifacts).toEqual([
+      {
+        scriptUrl: mapsAndEvents[0].scriptParsedEvent.url,
+        sourceMapUrl: 'http://www.example.com/path/bundle.js.map',
+        map: JSON.parse(mapsAndEvents[0].map),
+      },
+      {
+        scriptUrl: mapsAndEvents[1].scriptParsedEvent.url,
+        sourceMapUrl: 'http://www.example.com/bundle.js.map',
+        map: JSON.parse(mapsAndEvents[1].map),
+      },
+      {
+        scriptUrl: mapsAndEvents[2].scriptParsedEvent.url,
+        sourceMapUrl: mapsAndEvents[2].scriptParsedEvent.sourceMapURL,
+        map: JSON.parse(mapsAndEvents[2].map),
+      },
+    ]);
+  });
+
+  it('generates an error message when fetching map fails', async () => {
+    const mapsAndEvents = [
+      {
+        scriptParsedEvent: {
+          url: 'http://www.example.com/bundle.js',
+          sourceMapURL: 'http://www.example.com/bundle.js.map',
+        },
+        fetchError: 'Failed fetching source map',
+      },
+    ];
+    const artifact = await runSourceMaps(mapsAndEvents);
+    expect(artifact).toEqual([
+      {
+        scriptUrl: mapsAndEvents[0].scriptParsedEvent.url,
+        sourceMapUrl: mapsAndEvents[0].scriptParsedEvent.sourceMapURL,
+        errorMessage: 'Error: Failed fetching source map',
+        map: undefined,
+      },
+    ]);
+  });
+
+  it('generates an error message when map url cannot be resolved', async () => {
+    const mapsAndEvents = [
+      {
+        scriptParsedEvent: {
+          url: 'http://www.example.com/bundle.js',
+          sourceMapURL: 'http://',
+        },
+      },
+    ];
+    const artifact = await runSourceMaps(mapsAndEvents);
+    expect(artifact).toEqual([
+      {
+        scriptUrl: mapsAndEvents[0].scriptParsedEvent.url,
+        sourceMapUrl: undefined,
+        errorMessage: 'Could not resolve map url: http://',
+        map: undefined,
+      },
+    ]);
+  });
+
+  it('generates an error message when parsing map fails', async () => {
+    const mapsAndEvents = [
+      {
+        scriptParsedEvent: {
+          url: 'http://www.example.com/bundle.js',
+          sourceMapURL: 'http://www.example.com/bundle.js.map',
+        },
+        map: '{{}',
+      },
+      {
+        scriptParsedEvent: {
+          url: 'http://www.example.com/bundle-2.js',
+          sourceMapURL: makeJsonDataUrl('{};'),
+        },
+      },
+    ];
+    const artifact = await runSourceMaps(mapsAndEvents);
+    expect(artifact).toEqual([
+      {
+        scriptUrl: mapsAndEvents[0].scriptParsedEvent.url,
+        sourceMapUrl: mapsAndEvents[0].scriptParsedEvent.sourceMapURL,
+        errorMessage: 'SyntaxError: Unexpected token { in JSON at position 1',
+        map: undefined,
+      },
+      {
+        scriptUrl: mapsAndEvents[1].scriptParsedEvent.url,
+        sourceMapUrl: undefined,
+        errorMessage: 'SyntaxError: Unexpected token ; in JSON at position 2',
+        map: undefined,
+      },
+    ]);
+  });
+});
diff --git a/lighthouse-core/test/gather/mock-commands.js b/lighthouse-core/test/gather/mock-commands.js
new file mode 100644
index 000000000..32d787263
--- /dev/null
+++ b/lighthouse-core/test/gather/mock-commands.js
@@ -0,0 +1,119 @@
+/**
+ * @license Copyright 2019 Google Inc. All Rights Reserved.
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
+ */
+'use strict';
+
+/**
+ * @fileoverview Refer to driver-test.js and source-maps-test.js for intended usage.
+ */
+
+/* eslint-env jest */
+
+/**
+ * Creates a jest mock function whose implementation consumes mocked protocol responses matching the
+ * requested command in the order they were mocked.
+ *
+ * It is decorated with two methods:
+ *    - `mockResponse` which pushes protocol message responses for consumption
+ *    - `findInvocation` which asserts that `sendCommand` was invoked with the given command and
+ *      returns the protocol message argument.
+ */
+function createMockSendCommandFn() {
+  const mockResponses = [];
+  const mockFn = jest.fn().mockImplementation((command, ...args) => {
+    const indexOfResponse = mockResponses.findIndex(entry => entry.command === command);
+    if (indexOfResponse === -1) throw new Error(`${command} unimplemented`);
+    const {response, delay} = mockResponses[indexOfResponse];
+    mockResponses.splice(indexOfResponse, 1);
+    const returnValue = typeof response === 'function' ? response(...args) : response;
+    if (delay) return new Promise(resolve => setTimeout(() => resolve(returnValue), delay));
+    return Promise.resolve(returnValue);
+  });
+
+  mockFn.mockResponse = (command, response, delay) => {
+    mockResponses.push({command, response, delay});
+    return mockFn;
+  };
+
+  mockFn.findInvocation = command => {
+    expect(mockFn).toHaveBeenCalledWith(command, expect.anything());
+    return mockFn.mock.calls.find(call => call[0] === command)[1];
+  };
+
+  return mockFn;
+}
+
+/**
+ * Creates a jest mock function whose implementation invokes `.on`/`.once` listeners after a setTimeout tick.
+ * Closely mirrors `createMockSendCommandFn`.
+ *
+ * It is decorated with two methods:
+ *    - `mockEvent` which pushes protocol event payload for consumption
+ *    - `findListener` which asserts that `on` was invoked with the given event name and
+ *      returns the listener .
+ */
+function createMockOnceFn() {
+  const mockEvents = [];
+  const mockFn = jest.fn().mockImplementation((eventName, listener) => {
+    const indexOfResponse = mockEvents.findIndex(entry => entry.event === eventName);
+    if (indexOfResponse === -1) return;
+    const {response} = mockEvents[indexOfResponse];
+    mockEvents.splice(indexOfResponse, 1);
+    // Wait a tick because real events never fire immediately
+    setTimeout(() => listener(response), 0);
+  });
+
+  mockFn.mockEvent = (event, response) => {
+    mockEvents.push({event, response});
+    return mockFn;
+  };
+
+  mockFn.findListener = event => {
+    expect(mockFn).toHaveBeenCalledWith(event, expect.anything());
+    return mockFn.mock.calls.find(call => call[0] === event)[1];
+  };
+
+  return mockFn;
+}
+
+/**
+ * Very much like `createMockOnceFn`, but will fire all the events (not just one for every call).
+ * So it's good for .on w/ many events.
+ */
+function createMockOnFn() {
+  const mockEvents = [];
+  const mockFn = jest.fn().mockImplementation((eventName, listener) => {
+    const events = mockEvents.filter(entry => entry.event === eventName);
+    if (!events.length) return;
+    for (const event of events) {
+      const indexOfEvent = mockEvents.indexOf(event);
+      mockEvents.splice(indexOfEvent, 1);
+    }
+    // Wait a tick because real events never fire immediately
+    setTimeout(() => {
+      for (const event of events) {
+        listener(event.response);
+      }
+    }, 0);
+  });
+
+  mockFn.mockEvent = (event, response) => {
+    mockEvents.push({event, response});
+    return mockFn;
+  };
+
+  mockFn.findListener = event => {
+    expect(mockFn).toHaveBeenCalledWith(event, expect.anything());
+    return mockFn.mock.calls.find(call => call[0] === event)[1];
+  };
+
+  return mockFn;
+}
+
+module.exports = {
+  createMockSendCommandFn,
+  createMockOnceFn,
+  createMockOnFn,
+};