Repository: GoogleChrome/lighthouse
Pull request #12556 (2021-05-25 17:04:49)

core: update FCP score curve
Part of #11866

Updates the FCP score-curve control points to match the latest Lighthouse performance data and to align with other web performance tools.

Our FCP control points were last updated more than three years ago. It's not clear why FCP in HTTP Archive has trended lower over the time since (maybe changes in the sampled URLs?), but they clearly don't match the thresholds we would pick today.

Other tools are also trying to align on a set of FCP thresholds. At one point 1000ms/3000ms was recommended for FCP good/average thresholds (I believe that's what CrUX uses today), but more recent analysis has found the 1000ms to be difficult to be widely achievable, and so the current recommendation for tooling is 1800ms/3000ms.

While in some cases a more easily achievable threshold might be _too_ easy for Lighthouse, in this case it's such an improvement over the current speed curve that this seems like an excellent move.

In addition, if we were picking new control points without outside guidance, the May 2021 HTTP Archive run gives

p08 - 1787ms
p25 - 2595ms
(3000ms is closer to p35, but 2595ms is close enough that it seems ok)

With the [change in HTTP Archive throttling in mind](https://github.com/GoogleChromeLabs/lh-metrics-analysis/blob/gh-pages/reports/manual/2020-10-http-archive-throttling-vs-cli/report.md), we can also look at the July 2020 run, the month before the throttling change, and we find

p08 - 1662ms
P25 - 2484ms
(1800ms is closer to p10, 3000ms is closer to p40)

The old throttling is closer to what we see in CLI Lighthouse, so we could move to these even tougher numbers, but the proposed numbers are still a big improvement over the current control points and alignment would be ideal, so it seems worth moving to 1800/3000.


### Desktop
There's no web vitals guidance for desktop specifically, but we documented using `SELECT QUANTILES(renderStart, 21) FROM [httparchive:summary_pages.2018_12_15_desktop] LIMIT 1000` to derive the desktop control points. Running that today:

May 2021
p08 - 1000ms
p25 - 1500ms

July 2020
P08 - 1000ms
P25 - 1600ms

These match our current values (934/1600) very closely, so I suggest we leave them unchanged.


Patch:
diff --git a/lighthouse-core/audits/metrics/first-contentful-paint-3g.js b/lighthouse-core/audits/metrics/first-contentful-paint-3g.js
index 8daff84c5..768e0f7ef 100644
--- a/lighthouse-core/audits/metrics/first-contentful-paint-3g.js
+++ b/lighthouse-core/audits/metrics/first-contentful-paint-3g.js
@@ -29,12 +29,11 @@ class FirstContentfulPaint3G extends Audit {
    */
   static get defaultOptions() {
     return {
-      // 25th and 5th percentiles HTTPArchive on Fast 3G -> multiply by 1.5 for RTT differential -> median and PODR
-      // p10 is then derived from them.
-      // https://bigquery.cloud.google.com/table/httparchive:lighthouse.2018_04_01_mobile?pli=1
-      // https://www.desmos.com/calculator/fflcrsn9sj
-      p10: 3504,
-      median: 6000,
+      // 25th and 8th percentiles HTTPArchive on Slow 4G -> multiply by 1.5 for RTT differential -> median and p10.
+      // https://bigquery.cloud.google.com/table/httparchive:lighthouse.2021_05_01_mobile
+      // https://www.desmos.com/calculator/xi5oympawp
+      p10: 2700,
+      median: 4500,
     };
   }
 
diff --git a/lighthouse-core/audits/metrics/first-contentful-paint.js b/lighthouse-core/audits/metrics/first-contentful-paint.js
index 226c9b731..c45aadd98 100644
--- a/lighthouse-core/audits/metrics/first-contentful-paint.js
+++ b/lighthouse-core/audits/metrics/first-contentful-paint.js
@@ -37,16 +37,16 @@ class FirstContentfulPaint extends Audit {
   static get defaultOptions() {
     return {
       mobile: {
-        // 25th and 5th percentiles HTTPArchive -> median and PODR, then p10 is derived from them.
-        // https://bigquery.cloud.google.com/table/httparchive:lighthouse.2018_04_01_mobile?pli=1
-        // see https://www.desmos.com/calculator/oqlvmezbze
+        // 25th and 8th percentiles HTTPArchive -> median and p10.
+        // https://bigquery.cloud.google.com/table/httparchive:lighthouse.2021_05_01_mobile
+        // see https://www.desmos.com/calculator/6wi8rhipve
         scoring: {
-          p10: 2336,
-          median: 4000,
+          p10: 1800,
+          median: 3000,
         },
       },
       desktop: {
-        // SELECT QUANTILES(renderStart, 21) FROM [httparchive:summary_pages.2018_12_15_desktop] LIMIT 1000
+        // SELECT QUANTILES(renderStart, 21) FROM [httparchive:summary_pages.2020_07_01_desktop] LIMIT 1000
         scoring: {
           p10: 934,
           median: 1600,


Test patch:
diff --git a/lighthouse-core/test/audits/metrics/first-contentful-paint-3g-test.js b/lighthouse-core/test/audits/metrics/first-contentful-paint-3g-test.js
index ea41fe865..beae2bb98 100644
--- a/lighthouse-core/test/audits/metrics/first-contentful-paint-3g-test.js
+++ b/lighthouse-core/test/audits/metrics/first-contentful-paint-3g-test.js
@@ -28,7 +28,7 @@ describe('Performance: first-contentful-paint-3g audit', () => {
     // Use InlineSnapshot here so changes to Lantern coefficients can be easily updated en masse
     expect({score: result.score, value: Math.round(result.numericValue)}).toMatchInlineSnapshot(`
 Object {
-  "score": 0.99,
+  "score": 0.97,
   "value": 2087,
 }
 `);
diff --git a/lighthouse-core/test/audits/metrics/first-contentful-paint-test.js b/lighthouse-core/test/audits/metrics/first-contentful-paint-test.js
index 9ee471d1a..9e7604029 100644
--- a/lighthouse-core/test/audits/metrics/first-contentful-paint-test.js
+++ b/lighthouse-core/test/audits/metrics/first-contentful-paint-test.js
@@ -4,14 +4,17 @@
  * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
  */
 'use strict';
-const Audit = require('../../../audits/metrics/first-contentful-paint.js');
+const FcpAudit = require('../../../audits/metrics/first-contentful-paint.js');
 const assert = require('assert').strict;
-const options = Audit.defaultOptions;
+const options = FcpAudit.defaultOptions;
 const constants = require('../../../config/constants.js');
 
 const pwaTrace = require('../../fixtures/traces/progressive-app-m60.json');
 const pwaDevtoolsLog = require('../../fixtures/traces/progressive-app-m60.devtools.log.json');
 
+const frameTrace = require('../../fixtures/traces/frame-metrics-m90.json');
+const frameDevtoolsLog = require('../../fixtures/traces/frame-metrics-m90.devtools.log.json');
+
 /**
  * @param {{
  * {LH.SharedFlagsSettings['formFactor']} formFactor
@@ -35,16 +38,32 @@ describe('Performance: first-contentful-paint audit', () => {
   it('evaluates valid input correctly', async () => {
     const artifacts = {
       traces: {
-        [Audit.DEFAULT_PASS]: pwaTrace,
+        [FcpAudit.DEFAULT_PASS]: pwaTrace,
       },
       devtoolsLogs: {
-        [Audit.DEFAULT_PASS]: pwaDevtoolsLog,
+        [FcpAudit.DEFAULT_PASS]: pwaDevtoolsLog,
       },
     };
 
     const context = getFakeContext({formFactor: 'mobile', throttlingMethod: 'provided'});
-    const result = await Audit.audit(artifacts, context);
+    const result = await FcpAudit.audit(artifacts, context);
     assert.equal(result.score, 1);
     assert.equal(result.numericValue, 498.87);
   });
+
+  it('evaluates a modern trace correctly', async () => {
+    const artifacts = {
+      traces: {
+        [FcpAudit.DEFAULT_PASS]: frameTrace,
+      },
+      devtoolsLogs: {
+        [FcpAudit.DEFAULT_PASS]: frameDevtoolsLog,
+      },
+    };
+
+    const context = getFakeContext({formFactor: 'mobile', throttlingMethod: 'provided'});
+    const result = await FcpAudit.audit(artifacts, context);
+    assert.equal(result.score, 0.06);
+    assert.equal(result.numericValue, 5668.275);
+  });
 });
diff --git a/lighthouse-core/test/report/report-generator-test.js b/lighthouse-core/test/report/report-generator-test.js
index bcc649a83..90e4e7157 100644
--- a/lighthouse-core/test/report/report-generator-test.js
+++ b/lighthouse-core/test/report/report-generator-test.js
@@ -104,8 +104,8 @@ describe('ReportGenerator', () => {
       expect(lines.length).toBeGreaterThan(100);
       expect(lines.slice(0, 3).join('\n')).toMatchInlineSnapshot(`
         "requestedUrl,finalUrl,category,name,title,type,score
-        \\"http://localhost:10200/dobetterweb/dbw_tester.html\\",\\"http://localhost:10200/dobetterweb/dbw_tester.html\\",\\"Performance\\",\\"performance-score\\",\\"Overall Performance Category Score\\",\\"numeric\\",\\"0.64\\"
-        \\"http://localhost:10200/dobetterweb/dbw_tester.html\\",\\"http://localhost:10200/dobetterweb/dbw_tester.html\\",\\"Performance\\",\\"first-contentful-paint\\",\\"First Contentful Paint\\",\\"numeric\\",\\"0.51\\"
+        \\"http://localhost:10200/dobetterweb/dbw_tester.html\\",\\"http://localhost:10200/dobetterweb/dbw_tester.html\\",\\"Performance\\",\\"performance-score\\",\\"Overall Performance Category Score\\",\\"numeric\\",\\"0.6\\"
+        \\"http://localhost:10200/dobetterweb/dbw_tester.html\\",\\"http://localhost:10200/dobetterweb/dbw_tester.html\\",\\"Performance\\",\\"first-contentful-paint\\",\\"First Contentful Paint\\",\\"numeric\\",\\"0.24\\"
         "
       `);
 
diff --git a/lighthouse-core/test/results/sample_v2.json b/lighthouse-core/test/results/sample_v2.json
index 56c5ec99e..8d179d919 100644
--- a/lighthouse-core/test/results/sample_v2.json
+++ b/lighthouse-core/test/results/sample_v2.json
@@ -69,7 +69,7 @@
       "id": "first-contentful-paint",
       "title": "First Contentful Paint",
       "description": "First Contentful Paint marks the time at which the first text or image is painted. [Learn more](https://web.dev/first-contentful-paint/).",
-      "score": 0.51,
+      "score": 0.24,
       "scoreDisplayMode": "numeric",
       "numericValue": 3969.135,
       "numericUnit": "millisecond",
@@ -5277,7 +5277,7 @@
         }
       ],
       "id": "performance",
-      "score": 0.64
+      "score": 0.6
     },
     "accessibility": {
       "title": "Accessibility",