Repository: GoogleChrome/lighthouse
Pull request #10279 (2020-01-29 20:16:22)

misc: write lantern accuracy to file on update
**Bad News**
I couldn't replicate the accuracy stats from the screenshots in https://github.com/GoogleChrome/lighthouse/pull/10129#issue-355355302.

**Good News** 
LCP accuracy is actually fairly decent and I don't think we need any immediate or drastic action. The edge cases could certainly use some research and improvement but not enough to push out 6.0 IMO.

**This PR**
To make this easier to review and track moving forward, this PR writes the accuracy stats that appear in the screenshot to a file kept in git whenever the values are updated, so we know for certain how a PR affects the stats in aggregate without relying on screenshot comments (thanks @connorjclark for the idea!).

**Related Issues/PRs**
#10129



Patch:
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
index a93f10c48..8822f0f78 100644
--- a/.github/CODEOWNERS
+++ b/.github/CODEOWNERS
@@ -6,4 +6,4 @@
 
 
 # default owners for the entire repo
-*       @paulirish @connorjclark @patrickhulce @exterkamp
+*      @GoogleChrome/lighthouse-hackers
diff --git a/docs/authenticated-pages.md b/docs/authenticated-pages.md
index 469df26aa..49254fa17 100644
--- a/docs/authenticated-pages.md
+++ b/docs/authenticated-pages.md
@@ -8,6 +8,10 @@ Default runs of Lighthouse load a page as a "new user", with no previous session
 
 See [a working demo at /docs/recipes/auth](./recipes/auth).
 
+View our full documentation for using [Lighthouse along with Puppeteer](https://github.com/GoogleChrome/lighthouse/blob/master/docs/puppeteer.md).
+
+You may want to use Puppeteer's [`page.setCookie`](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagesetcookiecookies).
+
 ## Option 2: Leverage logged-in state with Chrome DevTools
 
 The Audits panel in Chrome DevTools will never clear your cookies, so you can log in to the target site and then run Lighthouse. If `localStorage` or `indexedDB` is important for your authentication purposes, be sure to uncheck `Clear storage`.
@@ -28,7 +32,7 @@ const result = await lighthouse('http://www.example.com', {
 });
 ```
 
-You could also set the `Cookie` header, but beware: it will [override any other Cookies you expect to be there](https://github.com/GoogleChrome/lighthouse/pull/9170). A workaround is to use Puppeteer's [`page.setCookie`](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagesetcookiecookies).
+You could also set the `Cookie` header, but beware: it will [override any other Cookies you expect to be there](https://github.com/GoogleChrome/lighthouse/pull/9170).
 
 ## Option 4: Open a debug instance of Chrome and manually log in
 
diff --git a/docs/puppeteer.md b/docs/puppeteer.md
index 95547832a..259be9dfd 100644
--- a/docs/puppeteer.md
+++ b/docs/puppeteer.md
@@ -1,9 +1,6 @@
-# Recipes Puppeteer with Lighthouse
+# Using Puppeteer with Lighthouse
 
-**Note**: https://github.com/GoogleChrome/lighthouse/issues/3837 tracks the discussion for making Lighthouse work in concert with Puppeteer.
-Some things are possible today (login to a page using Puppeteer, audit it using Lighthouse) while others (A/B testing the perf of UI changes) are trickier or not yet possible.
-
-### Inject JS/CSS before the page loads
+### Option 1: Launch Chrome with Puppeteer and handoff to Lighthouse
 
 The example below shows how to inject CSS into the page before Lighthouse audits the page.
 A similar approach can be taken for injecting JavaScript.
@@ -60,7 +57,7 @@ await browser.close();
 })();
 ```
 
-### Connecting Puppeteer to a browser instance launched by chrome-launcher.
+### Option 2: Launch Chrome with Lighthouse/chrome-launcher and handoff to Puppeteer
 
 When using Lighthouse programmatically, you'll often use chrome-launcher to launch Chrome.
 Puppeteer can reconnect to this existing browser instance like so:
@@ -100,3 +97,10 @@ await chrome.kill();
 
 })();
 ```
+
+### [Using Puppeteer in a custom gatherer](https://github.com/GoogleChrome/lighthouse/tree/master/docs/recipes/custom-gatherer-puppeteer)
+
+
+--------------
+
+**Note**: https://github.com/GoogleChrome/lighthouse/issues/3837 tracks the overall discussion for making Lighthouse work in concert with Puppeteer. Some things, like A/B testing the perf of UI changes, are tricky or not yet possible.
diff --git a/lighthouse-core/scripts/lantern/assert-master-lantern-values-unchanged.js b/lighthouse-core/scripts/lantern/assert-master-lantern-values-unchanged.js
index 623abd77b..a1561a29a 100755
--- a/lighthouse-core/scripts/lantern/assert-master-lantern-values-unchanged.js
+++ b/lighthouse-core/scripts/lantern/assert-master-lantern-values-unchanged.js
@@ -9,6 +9,7 @@
 /* eslint-disable no-console */
 
 const fs = require('fs');
+const assert = require('assert');
 const path = require('path');
 const constants = require('./constants.js');
 const chalk = require('chalk').default;
@@ -16,6 +17,7 @@ const chalk = require('chalk').default;
 const INPUT_PATH = process.argv[2] || constants.SITE_INDEX_WITH_GOLDEN_WITH_COMPUTED_PATH;
 const HEAD_PATH = path.resolve(process.cwd(), INPUT_PATH);
 const MASTER_PATH = constants.MASTER_COMPUTED_PATH;
+const MASTER_ACCURACY_PATH = constants.MASTER_ACCURACY_PATH;
 
 if (!fs.existsSync(HEAD_PATH) || !fs.existsSync(MASTER_PATH)) {
   throw new Error('Usage $0 <computed file>');
@@ -63,6 +65,10 @@ if (diffs.length) {
 
   process.exit(1);
 } else {
+  assert.deepStrictEqual(
+    constants.evaluateAllMetrics(computedResults, expectedResults),
+    require(MASTER_ACCURACY_PATH)
+  );
   console.log('✅  PASS    No changes between expected and computed!');
 }
 
diff --git a/lighthouse-core/scripts/lantern/constants.js b/lighthouse-core/scripts/lantern/constants.js
index 01901ea49..a669cd3f2 100644
--- a/lighthouse-core/scripts/lantern/constants.js
+++ b/lighthouse-core/scripts/lantern/constants.js
@@ -9,8 +9,190 @@ const path = require('path');
 
 /* eslint-disable max-len */
 
+// TODO: If we need to touch these again, it's time to make lantern.d.ts :)
+// @see https://github.com/GoogleChrome/lighthouse/pull/10279#discussion_r373116456
+
+/**
+ * @typedef LanternSiteDefinition
+ * @property {string} url
+ * @property {TargetMetrics} wpt3g
+ * @property {LanternMetrics} lantern
+ * @property {LanternMetrics} [baseline]
+ */
+
+/**
+ * @typedef LanternEvaluation
+ * @property {string} url
+ * @property {string} metric
+ * @property {string} lanternMetric
+ * @property {number} expected
+ * @property {number} actual
+ * @property {number} diff
+ * @property {number} diffAsPercent
+ */
+
+/**
+ * @typedef EstimateEvaluationSummary
+ * @property {LanternEvaluation[]} evaluations
+ * @property {number} p50
+ * @property {number} p90
+ * @property {number} p95
+ */
+
+/**
+ * @typedef TargetMetrics
+ * @property {number} [firstContentfulPaint]
+ * @property {number} [firstMeaningfulPaint]
+ * @property {number} [timeToFirstInteractive]
+ * @property {number} [timeToConsistentlyInteractive]
+ * @property {number} [speedIndex]
+ * @property {number} [largestContentfulPaint]
+ */
+
+/**
+ * @typedef LanternMetrics
+ * @property {number} optimisticFCP
+ * @property {number} optimisticFMP
+ * @property {number} optimisticSI
+ * @property {number} optimisticTTFCPUI
+ * @property {number} optimisticTTI
+ * @property {number} optimisticLCP
+ * @property {number} pessimisticFCP
+ * @property {number} pessimisticFMP
+ * @property {number} pessimisticSI
+ * @property {number} pessimisticTTFCPUI
+ * @property {number} pessimisticTTI
+ * @property {number} pessimisticLCP
+ * @property {number} roughEstimateOfFCP
+ * @property {number} roughEstimateOfFMP
+ * @property {number} roughEstimateOfSI
+ * @property {number} roughEstimateOfTTFCPUI
+ * @property {number} roughEstimateOfTTI
+ * @property {number} roughEstimateOfLCP
+ */
+
+/** @type {Array<string>} */
+const WARNINGS = [];
+
 module.exports = {
+  WARNINGS,
+  // prettier-ignore
   SITE_INDEX_WITH_GOLDEN_PATH: './lantern-data/site-index-plus-golden-expectations.json',
+  // prettier-ignore
   SITE_INDEX_WITH_GOLDEN_WITH_COMPUTED_PATH: path.join(__dirname, '../../../.tmp/site-index-plus-golden-expectations-plus-computed.json'),
+  // prettier-ignore
   MASTER_COMPUTED_PATH: path.join(__dirname, '../../test/fixtures/lantern-master-computed-values.json'),
+  // prettier-ignore
+  MASTER_ACCURACY_PATH: path.join(__dirname, '../../test/fixtures/lantern-master-accuracy.json'),
+  /**
+   * @param {{sites: Array<LanternSiteDefinition>}} siteIndexWithComputed
+   * @param {{sites: Array<LanternMetrics & {url: string}>}} baselineLanternData
+   */
+  combineBaselineAndComputedDatasets(siteIndexWithComputed, baselineLanternData) {
+    for (const site of baselineLanternData.sites) {
+      const computedSite = siteIndexWithComputed.sites.find(entry => entry.url === site.url);
+      if (!computedSite) continue;
+      computedSite.baseline = site;
+    }
+
+    const entries = siteIndexWithComputed.sites.filter(site => site.lantern && site.baseline);
+
+    if (!entries.length) {
+      throw new Error('No lantern metrics available, did you run run-on-all-assets.js?');
+    }
+
+    return entries;
+  },
+
+  /**
+   * @param {LanternSiteDefinition} site
+   * @param {TargetMetrics} expectedMetrics
+   * @param {LanternMetrics} actualMetrics
+   * @param {keyof TargetMetrics} metric
+   * @param {keyof LanternMetrics} lanternMetric
+   * @return {(LanternEvaluation & LanternSiteDefinition)|null}
+   */
+  evaluateSite(site, expectedMetrics, actualMetrics, metric, lanternMetric) {
+    const expected = Math.round(expectedMetrics[metric]);
+    if (expected === 0) return null;
+
+    const actual = Math.round(actualMetrics[lanternMetric]);
+    const diff = Math.abs(actual - expected);
+    const diffAsPercent = diff / expected;
+
+    return {
+      ...site,
+      expected,
+      actual,
+      diff,
+      diffAsPercent,
+      metric,
+      lanternMetric,
+    };
+  },
+
+  /**
+   * @param {LanternSiteDefinition[]} entries
+   * @param {keyof TargetMetrics} metric
+   * @param {keyof LanternMetrics} lanternMetric
+   * @param {'lantern'|'baseline'} [lanternOrBaseline]
+   * @return {EstimateEvaluationSummary}
+   */
+  evaluateAccuracy(entries, metric, lanternMetric, lanternOrBaseline = 'lantern') {
+    const evaluations = [];
+
+    const percentErrors = [];
+    for (const entry of entries) {
+      const evaluation = this.evaluateSite(
+        entry,
+        entry.wpt3g,
+        entry[lanternOrBaseline],
+        metric,
+        lanternMetric
+      );
+
+      // No data was available at all, skip it.
+      if (!evaluation) continue;
+
+      // Data was supposed to be available, but one metric was missing, warn.
+      if (!Number.isFinite(evaluation.diff)) {
+        const missingMetric = Number.isFinite(entry.wpt3g[metric]) ? lanternMetric : metric;
+        const message = `WARNING: ${evaluation.url} was missing values for ${missingMetric}`;
+        WARNINGS.push(message);
+        continue;
+      }
+
+      evaluations.push(evaluation);
+      percentErrors.push(evaluation.diffAsPercent);
+    }
+
+    percentErrors.sort((a, b) => a - b);
+
+    const p50 = percentErrors[Math.floor((percentErrors.length / 100) * 50)];
+    const p90 = percentErrors[Math.floor((percentErrors.length / 100) * 90)];
+    const p95 = percentErrors[Math.floor((percentErrors.length / 100) * 95)];
+    return {evaluations, p50, p90, p95};
+  },
+  /**
+   * @param {{sites: Array<LanternSiteDefinition>}} siteIndexWithComputed
+   * @param {{sites: Array<LanternMetrics & {url: string}>}} baselineData
+   */
+  evaluateAllMetrics(siteIndexWithComputed, baselineData) {
+    const entries = this.combineBaselineAndComputedDatasets(siteIndexWithComputed, baselineData);
+    /** @param {keyof TargetMetrics} metric @param {keyof LanternMetrics} lanternMetric */
+    const evaluate = (metric, lanternMetric) => {
+      const result = this.evaluateAccuracy(entries, metric, lanternMetric);
+      delete result.evaluations;
+      return result;
+    };
+
+    return {
+      roughEstimateOfFCP: evaluate('firstContentfulPaint', 'roughEstimateOfFCP'),
+      roughEstimateOfFMP: evaluate('firstMeaningfulPaint', 'roughEstimateOfFMP'),
+      roughEstimateOfSI: evaluate('speedIndex', 'roughEstimateOfSI'),
+      roughEstimateOfTTFCPUI: evaluate('timeToFirstInteractive', 'roughEstimateOfTTFCPUI'),
+      roughEstimateOfTTI: evaluate('timeToConsistentlyInteractive', 'roughEstimateOfTTI'),
+      roughEstimateOfLCP: evaluate('largestContentfulPaint', 'roughEstimateOfLCP'),
+    };
+  },
 };
diff --git a/lighthouse-core/scripts/lantern/print-correlations.js b/lighthouse-core/scripts/lantern/print-correlations.js
index 79dc11f3d..fcbf18a9d 100755
--- a/lighthouse-core/scripts/lantern/print-correlations.js
+++ b/lighthouse-core/scripts/lantern/print-correlations.js
@@ -8,64 +8,11 @@
 
 /* eslint-disable no-console */
 
-/**
- * @typedef LanternSiteDefinition
- * @property {string} url
- * @property {TargetMetrics} wpt3g
- * @property {LanternMetrics} lantern
- * @property {LanternMetrics} baseline
- */
-
-/**
- * @typedef LanternEvaluation
- * @property {string} url
- * @property {string} metric
- * @property {string} lanternMetric
- * @property {number} expected
- * @property {number} actual
- * @property {number} diff
- * @property {number} diffAsPercent
- */
-
-/**
- * @typedef EstimateEvaluationSummary
- * @property {LanternEvaluation[]} evaluations
- * @property {number} p50
- * @property {number} p90
- * @property {number} p95
- */
-
-/**
- * @typedef TargetMetrics
- * @property {number} [firstContentfulPaint]
- * @property {number} [firstMeaningfulPaint]
- * @property {number} [timeToFirstInteractive]
- * @property {number} [timeToConsistentlyInteractive]
- * @property {number} [speedIndex]
- * @property {number} [largestContentfulPaint]
- */
-
-/**
- * @typedef LanternMetrics
- * @property {number} optimisticFCP
- * @property {number} optimisticFMP
- * @property {number} optimisticSI
- * @property {number} optimisticTTFCPUI
- * @property {number} optimisticTTI
- * @property {number} optimisticLCP
- * @property {number} pessimisticFCP
- * @property {number} pessimisticFMP
- * @property {number} pessimisticSI
- * @property {number} pessimisticTTFCPUI
- * @property {number} pessimisticTTI
- * @property {number} pessimisticLCP
- * @property {number} roughEstimateOfFCP
- * @property {number} roughEstimateOfFMP
- * @property {number} roughEstimateOfSI
- * @property {number} roughEstimateOfTTFCPUI
- * @property {number} roughEstimateOfTTI
- * @property {number} roughEstimateOfLCP
- */
+/** @typedef {import('./constants').LanternSiteDefinition} LanternSiteDefinition */
+/** @typedef {import('./constants').LanternEvaluation} LanternEvaluation */
+/** @typedef {import('./constants').EstimateEvaluationSummary} EstimateEvaluationSummary */
+/** @typedef {import('./constants').TargetMetrics} TargetMetrics */
+/** @typedef {import('./constants').LanternMetrics} LanternMetrics */
 
 const fs = require('fs');
 const path = require('path');
@@ -79,42 +26,15 @@ const INPUT_PATH = process.argv[2] || constants.SITE_INDEX_WITH_GOLDEN_WITH_COMP
 const COMPUTATIONS_PATH = path.resolve(process.cwd(), INPUT_PATH);
 const BASELINE_PATH = constants.MASTER_COMPUTED_PATH;
 
+
 if (!fs.existsSync(COMPUTATIONS_PATH)) throw new Error('Usage $0 <computed summary file>');
 
 /** @type {{sites: LanternSiteDefinition[]}} */
 const siteIndexWithComputed = require(COMPUTATIONS_PATH);
 const baselineLanternData = require(BASELINE_PATH);
 
-for (const site of baselineLanternData.sites) {
-  const computedSite = siteIndexWithComputed.sites.find(entry => entry.url === site.url);
-  if (!computedSite) continue;
-  computedSite.baseline = site;
-}
-
-const entries = siteIndexWithComputed.sites.filter(site => site.lantern && site.baseline);
-
-if (!entries.length) {
-  throw new Error('No lantern metrics available, did you run run-on-all-assets.js?');
-}
-
-/**
- * @param {LanternSiteDefinition} site
- * @param {TargetMetrics} expectedMetrics
- * @param {LanternMetrics} actualMetrics
- * @param {keyof TargetMetrics} metric
- * @param {keyof LanternMetrics} lanternMetric
- * @return {(LanternEvaluation & LanternSiteDefinition)|null}
- */
-function evaluateSite(site, expectedMetrics, actualMetrics, metric, lanternMetric) {
-  const expected = Math.round(expectedMetrics[metric]);
-  if (expected === 0) return null;
-
-  const actual = Math.round(actualMetrics[lanternMetric]);
-  const diff = Math.abs(actual - expected);
-  const diffAsPercent = diff / expected;
-
-  return {...site, expected, actual, diff, diffAsPercent, metric, lanternMetric};
-}
+const entries = constants.combineBaselineAndComputedDatasets(siteIndexWithComputed,
+  baselineLanternData);
 
 /** @param {LanternEvaluation} evaluation */
 function isEvaluationGood(evaluation) {
@@ -132,39 +52,6 @@ function isEvaluationBad(evaluation) {
   return evaluation.diffAsPercent > OK_DIFF_AS_PERCENT_THRESHOLD;
 }
 
-/**
- * @param {LanternSiteDefinition[]} entries
- * @param {keyof TargetMetrics} metric
- * @param {keyof LanternMetrics} lanternMetric
- * @param {'lantern'|'baseline'} [lanternOrBaseline]
- * @return {EstimateEvaluationSummary}
- */
-function evaluateAccuracy(entries, metric, lanternMetric, lanternOrBaseline = 'lantern') {
-  const evaluations = [];
-
-  const percentErrors = [];
-  for (const entry of entries) {
-    const evaluation = evaluateSite(
-      entry,
-      entry.wpt3g,
-      entry[lanternOrBaseline],
-      metric,
-      lanternMetric
-    );
-    if (!evaluation) continue;
-
-    evaluations.push(evaluation);
-    percentErrors.push(evaluation.diffAsPercent);
-  }
-
-  percentErrors.sort((a, b) => a - b);
-
-  const p50 = percentErrors[Math.floor((percentErrors.length / 100) * 50)];
-  const p90 = percentErrors[Math.floor((percentErrors.length / 100) * 90)];
-  const p95 = percentErrors[Math.floor((percentErrors.length / 100) * 95)];
-  return {evaluations, p50, p90, p95};
-}
-
 /** @type {LanternEvaluation[]} */
 const allEvaluations = [];
 /** @type {LanternEvaluation[]} */
@@ -207,8 +94,8 @@ function toPercentString(percentAsDecimal) {
  * @param {keyof LanternMetrics} lanternMetric
  */
 function evaluateAndPrintAccuracy(metric, lanternMetric) {
-  const actualAccuracy = evaluateAccuracy(entries, metric, lanternMetric);
-  const baselineAccuracy = evaluateAccuracy(entries, metric, lanternMetric, 'baseline');
+  const actualAccuracy = constants.evaluateAccuracy(entries, metric, lanternMetric);
+  const baselineAccuracy = constants.evaluateAccuracy(entries, metric, lanternMetric, 'baseline');
   const baselineOptions = {alwaysGray: !lanternMetric.includes('roughEstimate')};
 
   const strings = [
@@ -436,3 +323,10 @@ console.log(chalk.bold('\n ------- % Error Summary -------'));
 printPercentile(50);
 printPercentile(90);
 printPercentile(95);
+
+if (constants.WARNINGS.length) {
+  console.log('\n');
+  for (const message of new Set(constants.WARNINGS)) {
+    console.warn(chalk.yellowBright(message));
+  }
+}
diff --git a/lighthouse-core/scripts/lantern/update-master-lantern-values.js b/lighthouse-core/scripts/lantern/update-master-lantern-values.js
index 9cd48c59e..598a966da 100755
--- a/lighthouse-core/scripts/lantern/update-master-lantern-values.js
+++ b/lighthouse-core/scripts/lantern/update-master-lantern-values.js
@@ -19,6 +19,7 @@ const SITE_INDEX_PATH = path.resolve(process.cwd(), INPUT_PATH);
 const HEAD_COMPUTED_PATH = constants.SITE_INDEX_WITH_GOLDEN_WITH_COMPUTED_PATH;
 const RUN_ALL_SCRIPT_PATH = path.join(__dirname, 'run-on-all-assets.js');
 const OUTPUT_PATH = constants.MASTER_COMPUTED_PATH;
+const OUTPUT_ACCURACY_PATH = constants.MASTER_ACCURACY_PATH;
 
 if (!fs.existsSync(HEAD_COMPUTED_PATH) || process.env.FORCE) {
   if (!fs.existsSync(SITE_INDEX_PATH)) throw new Error('Usage $0 <expectations file>');
@@ -41,4 +42,10 @@ fs.writeFileSync(OUTPUT_PATH, prettyJSONStringify({sites}, {
   shouldExpand: (_, level) => level < 2,
 }));
 
+fs.writeFileSync(OUTPUT_ACCURACY_PATH, JSON.stringify(
+  constants.evaluateAllMetrics(computedResults, {sites}),
+  null,
+  2
+));
+
 
diff --git a/readme.md b/readme.md
index dfffe13f7..8ae164034 100644
--- a/readme.md
+++ b/readme.md
@@ -2,6 +2,32 @@
 
 > Lighthouse analyzes web apps and web pages, collecting modern performance metrics and insights on developer best practices.
 
+
+- Using Lighthouse
+  - [Using Lighthouse in Chrome DevTools](#using-lighthouse-in-chrome-devtools)
+  - [Using the Chrome extension](#using-the-chrome-extension)
+  - [Using the Node CLI](#using-the-node-cli)
+    * [CLI options](#cli-options)
+  - [Using the Node module](#using-the-node-module)
+- [Viewing a report](#viewing-a-report)
+  * [Online Viewer](#online-viewer)
+- [Docs & Recipes](#docs--recipes)
+- [Developing Lighthouse](#develop)
+  * [Setup](#setup)
+  * [Run](#run)
+  * [Tests](#tests)
+- Associated Products and Projects
+  - [Lighthouse Integrations](#lighthouse-integrations)
+  - [Plugins](#plugins)
+  - [Related Projects](#related-projects)
+- [FAQ](#faq)
+  * [How does Lighthouse work?](#how-does-lighthouse-work)
+  * [Can I configure the lighthouse run?](#can-i-configure-the-lighthouse-run)
+  * [How does Lighthouse use network throttling, and how can I make it better?](#how-does-lighthouse-use-network-throttling-and-how-can-i-make-it-better)
+  * [Are results sent to a remote server?](#are-results-sent-to-a-remote-server)
+  * [How do I author custom audits to extend Lighthouse?](#how-do-i-author-custom-audits-to-extend-lighthouse)
+  * [How do I contribute?](#how-do-i-contribute)
+
 ## Using Lighthouse in Chrome DevTools
 
 Lighthouse is integrated directly into the Chrome Developer Tools, under the "Audits" panel.
diff --git a/tsconfig.json b/tsconfig.json
index d660f4c86..70f307899 100644
--- a/tsconfig.json
+++ b/tsconfig.json
@@ -28,6 +28,7 @@
   ],
   "files": [
     // Opt-in to typechecking for some core tests.
+    "lighthouse-core/test/scripts/lantern/constants-test.js",
     "lighthouse-core/test/gather/driver-test.js",
     "lighthouse-core/test/gather/gather-runner-test.js",
   ],


Test patch:
diff --git a/lighthouse-core/test/fixtures/lantern-master-accuracy.json b/lighthouse-core/test/fixtures/lantern-master-accuracy.json
new file mode 100644
index 000000000..dfcfef02a
--- /dev/null
+++ b/lighthouse-core/test/fixtures/lantern-master-accuracy.json
@@ -0,0 +1,32 @@
+{
+  "roughEstimateOfFCP": {
+    "p50": 0.292391744233104,
+    "p90": 0.5016256386437529,
+    "p95": 0.5795269168026101
+  },
+  "roughEstimateOfFMP": {
+    "p50": 0.3129196884233145,
+    "p90": 0.5385564466378778,
+    "p95": 0.6641924269472893
+  },
+  "roughEstimateOfSI": {
+    "p50": 0.27011225444340503,
+    "p90": 0.6267744048918978,
+    "p95": 0.9847542003733665
+  },
+  "roughEstimateOfTTFCPUI": {
+    "p50": 0.307951137444225,
+    "p90": 0.6159518516510329,
+    "p95": 0.7404564989707104
+  },
+  "roughEstimateOfTTI": {
+    "p50": 0.3131914893617021,
+    "p90": 0.6774419418835337,
+    "p95": 0.7467400959317818
+  },
+  "roughEstimateOfLCP": {
+    "p50": 0.2813953488372093,
+    "p90": 0.9142573930282505,
+    "p95": 1.6544450025113009
+  }
+}
\ No newline at end of file
diff --git a/lighthouse-core/test/scripts/lantern/constants-test.js b/lighthouse-core/test/scripts/lantern/constants-test.js
new file mode 100644
index 000000000..df4fe97d5
--- /dev/null
+++ b/lighthouse-core/test/scripts/lantern/constants-test.js
@@ -0,0 +1,194 @@
+/**
+ * @license Copyright 2020 Google Inc. All Rights Reserved.
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
+ */
+'use strict';
+
+const constants = require('../../../scripts/lantern/constants.js');
+
+/* eslint-env jest */
+
+/** @typedef {import('../../../scripts/lantern/constants.js').LanternSiteDefinition} LanternSite */
+
+describe('Lantern script helpers', () => {
+  /** @param {{url?: string, wpt3g: number, lantern: number}} param @return {LanternSite} */
+  const getSite = ({url, lantern, wpt3g}) => ({
+    url: url || 'http://example.com',
+    wpt3g: {firstContentfulPaint: wpt3g},
+    lantern: /** @type {LanternSite['lantern']} */ ({roughEstimateOfFCP: lantern}),
+  });
+
+  /** @param {{url: string, value: number}} param */
+  const getBaseline = ({url, value}) => /** @type {LanternSite['lantern'] & {url: string}} */ ({
+    url,
+    roughEstimateOfFCP: value,
+  });
+
+  describe('#combineBaselineAndComputedDatasets', () => {
+    it('should combine the two datasets on URL', () => {
+      const siteIndex = {
+        sites: [
+          getSite({url: 'urlA', wpt3g: 150, lantern: 100}),
+          getSite({url: 'urlB', wpt3g: 150, lantern: 100}),
+        ],
+      };
+      const baseline = {
+        sites: [getBaseline({url: 'urlA', value: 100}), getBaseline({url: 'urlC', value: 100})],
+      };
+
+      expect(constants.combineBaselineAndComputedDatasets(siteIndex, baseline)).toEqual([
+        {
+          url: 'urlA',
+          wpt3g: {firstContentfulPaint: 150},
+          lantern: {roughEstimateOfFCP: 100},
+          baseline: {roughEstimateOfFCP: 100, url: 'urlA'},
+        },
+      ]);
+    });
+
+    it('should throw on empty data', () => {
+      const siteIndex = {
+        sites: [getSite({url: 'urlB', wpt3g: 150, lantern: 100})],
+      };
+      const baseline = {
+        sites: [getBaseline({url: 'urlA', value: 100}), getBaseline({url: 'urlC', value: 100})],
+      };
+
+      expect(() => constants.combineBaselineAndComputedDatasets(siteIndex, baseline)).toThrow();
+    });
+  });
+
+  describe('#evaluateSite', () => {
+    it('should compute the diff', () => {
+      const site = getSite({url: 'http://example.com', wpt3g: 1000, lantern: 900});
+
+      expect(
+        constants.evaluateSite(
+          site,
+          site.wpt3g,
+          site.lantern,
+          'firstContentfulPaint',
+          'roughEstimateOfFCP'
+        )
+      ).toEqual({
+        url: 'http://example.com',
+        expected: 1000,
+        actual: 900,
+        diff: 100,
+        diffAsPercent: 0.1,
+        metric: 'firstContentfulPaint',
+        lanternMetric: 'roughEstimateOfFCP',
+        lantern: site.lantern,
+        wpt3g: site.wpt3g,
+      });
+    });
+
+    it('should return null on 0-data', () => {
+      const site = getSite({url: 'http://example.com', wpt3g: 0, lantern: 900});
+
+      expect(
+        constants.evaluateSite(
+          site,
+          site.wpt3g,
+          site.lantern,
+          'firstContentfulPaint',
+          'roughEstimateOfFCP'
+        )
+      ).toEqual(null);
+    });
+  });
+
+  describe('#evaluateAccuracy', () => {
+    /** @param {Array<{url?: string, wpt3g: number, lantern: number}>} entries */
+    const evaluate = entries =>
+      constants.evaluateAccuracy(
+        entries.map(getSite),
+        'firstContentfulPaint',
+        'roughEstimateOfFCP'
+      );
+
+    it('should build evaluations', () => {
+      const entries = [
+        {url: 'pageA', lantern: 1000, wpt3g: 2000},
+        {url: 'pageB', lantern: 3000, wpt3g: 2000},
+        {url: 'pageC', lantern: 5000, wpt3g: 1000},
+        {url: 'pageD', lantern: 1000, wpt3g: 1000},
+      ];
+
+      expect(evaluate(entries)).toEqual({
+        p50: 0.5,
+        p90: 4,
+        p95: 4,
+        evaluations: [
+          {
+            actual: 1000,
+            diff: 1000,
+            diffAsPercent: 0.5,
+            expected: 2000,
+            lantern: {roughEstimateOfFCP: 1000},
+            lanternMetric: 'roughEstimateOfFCP',
+            metric: 'firstContentfulPaint',
+            url: 'pageA',
+            wpt3g: {firstContentfulPaint: 2000},
+          },
+          {
+            actual: 3000,
+            diff: 1000,
+            diffAsPercent: 0.5,
+            expected: 2000,
+            lantern: {roughEstimateOfFCP: 3000},
+            lanternMetric: 'roughEstimateOfFCP',
+            metric: 'firstContentfulPaint',
+            url: 'pageB',
+            wpt3g: {firstContentfulPaint: 2000},
+          },
+          {
+            actual: 5000,
+            diff: 4000,
+            diffAsPercent: 4,
+            expected: 1000,
+            lantern: {roughEstimateOfFCP: 5000},
+            lanternMetric: 'roughEstimateOfFCP',
+            metric: 'firstContentfulPaint',
+            url: 'pageC',
+            wpt3g: {firstContentfulPaint: 1000},
+          },
+          {
+            actual: 1000,
+            diff: 0,
+            diffAsPercent: 0,
+            expected: 1000,
+            lantern: {roughEstimateOfFCP: 1000},
+            lanternMetric: 'roughEstimateOfFCP',
+            metric: 'firstContentfulPaint',
+            url: 'pageD',
+            wpt3g: {firstContentfulPaint: 1000},
+          },
+        ],
+      });
+    });
+
+    it('should compute p90 and p95 correctly', () => {
+      const entries = [];
+      for (let i = 0; i < 100; i++) {
+        // Construct a test array of 100 items where the error percent for each is equal to its index
+        // i=0, lantern=1000, wpt3g=1000, error=0, errorPercent=0
+        // i=1, lantern=2000, wpt3g=1000, error=1000, errorPercent=1
+        // i=2, lantern=3000, wpt3g=1000, error=2000, errorPercent=2
+        // ...
+        entries.push({lantern: (i + 1) * 1000, wpt3g: 1000});
+      }
+
+      // Shuffle the array to make sure it's sorting
+      entries.sort(() => Math.random() - 0.5);
+
+      const data = evaluate(entries);
+      expect(data).toMatchObject({
+        p50: 50,
+        p90: 90,
+        p95: 95,
+      });
+    });
+  });
+});