Repository: GoogleChrome/lighthouse (js)

Issue:
Evolve the next-generation image formats audit (AVIF)
**tl;dr: Could we update the next-gen formats audit to provide AVIF savings estimates?** 

**Background**

WebP has been around for a number of years now with support recently hitting 92% of browsers worldwide. In the years since it was introduced, another promising format has landed in browsers - [AVIF](https://jakearchibald.com/2020/avif-has-landed/) (supported in [Chromium](https://caniuse.com/avif)) (support is 62% globally), and coming to [Firefox 88-90](https://9to5linux.com/firefox-88-beta-adds-smooth-pinch-zooming-support-for-linux-on-wayland-enables-avif-by-default).  AVIF offers up to a 50% improvement in compression over JPEG and some estimates suggest a [20%](https://reachlightspeed.com/blog/using-the-new-high-performance-avif-image-format-on-the-web-today/) improvement over WebP. 

Some developers are seeing savings from switching to AVIF of up to [25%](https://twitter.com/TimVereecke/status/1372210487191085063) (based on payload bytes) with up to a 90ms [saving](https://twitter.com/TimVereecke/status/1372228787866329091) to LCP at P75. 

**Current Lighthouse audit**

Our current `serve images in next-gen formats` audit [source](https://github.com/GoogleChrome/lighthouse/blob/1ca0e7f1fddf73c8ec76a9e48db2546c0aa06c67/lighthouse-core/audits/byte-efficiency/uses-webp-images.js) estimates potential savings based on conversion figures for JPEG compression ratios (8:1) and WebP (10:1). 

![image](https://user-images.githubusercontent.com/110953/112392625-f8bcee80-8cb6-11eb-862c-f47d01d3faf9.png)

**Proposal**

What if we evolved this approach to either:

1. Replace the current WebP savings estimates with AVIF savings estimates? OR
2. Display two potential savings columns: one for WebP and one for AVIF? (i.e encourage at least serving what most browsers support, but if you're already using `<picture>`, consider adding AVIF as a source too)

| URL           | Size | Potential savings: WebP | AVIF |
|---------------|------|-------------------------|------|
| cool-cats.jpg | 75KB |                    30KB | 50KB |

**What are the potential savings estimates for AVIF?**

The AVIF team at Google have confidence in the 50% savings number (compared to JPEG) being accurate and the one we should use for tooling purposes (as of today) if we need a multiplier for estimates. Further evidence to back this number up includes:

* [Netflix study on AVIF](https://netflixtechblog.com/avif-for-next-generation-image-coding-b1d75675fe4)
* [Netflix presentation given to the Alliance for Open Media in 2019](http://aomedia.org/research/symposium-2019/#av1-image-file-format-avif-14)
* Next quarter, they'll have a dedicated test framework for comparisons that may allow us to refine the savings estimates in the future, but for now the 50 number is fine to use.

https://github.com/GoogleChrome/lighthouse/blob/1ca0e7f1fddf73c8ec76a9e48db2546c0aa06c67/lighthouse-core/audits/byte-efficiency/uses-webp-images.js#L56-L65

I think encouraging use of AVIF could help developers improve their Largest Contentful Paint when its an image.  This may also give us a chance to revise the current audit text to include AVIF generally.

Would love to hear what folks think. cc @paulirish @connorjclark @patrickhulce 

Pull request:
core(modern-images): update to include AVIF estimates
**Summary**
Updates the modern image formats audit to use AVIF file size estimates according to the plan outlined in https://github.com/GoogleChrome/lighthouse/issues/12295#issuecomment-823558270. I initially tried out showing both the WebP and AVIF estimates in the table, but a few problems:

- Very crowded with the "Resource Size" already there as well.
- The extra number provides basically no additional information since one is basically a simple multiple of the other. 

Instead, this PR just replaces the estimate used for the audit results with the AVIF estimate, skips an image if it's already WebP, and moves the WebP estimate to a new property `wastedWebpBytes`.

**Related Issues/PRs**
closes #12295


Patch:
diff --git a/lighthouse-core/audits/byte-efficiency/modern-image-formats.js b/lighthouse-core/audits/byte-efficiency/modern-image-formats.js
index 926adf540..770407efc 100644
--- a/lighthouse-core/audits/byte-efficiency/modern-image-formats.js
+++ b/lighthouse-core/audits/byte-efficiency/modern-image-formats.js
@@ -16,7 +16,7 @@ const UIStrings = {
   /** Imperative title of a Lighthouse audit that tells the user to serve images in newer and more efficient image formats in order to enhance the performance of a page. A non-modern image format was designed 20+ years ago. This is displayed in a list of audit titles that Lighthouse generates. */
   title: 'Serve images in next-gen formats',
   /** Description of a Lighthouse audit that tells the user *why* they should use newer and more efficient image formats. This is displayed after a user expands the section to see more. No character length limits. 'Learn More' becomes link text to additional documentation. */
-  description: 'Image formats like JPEG 2000, JPEG XR, and WebP often provide better ' +
+  description: 'Image formats like WebP and AVIF often provide better ' +
     'compression than PNG or JPEG, which means faster downloads and less data consumption. ' +
     '[Learn more](https://web.dev/uses-webp-images/).',
 };
@@ -40,16 +40,6 @@ class ModernImageFormats extends ByteEfficiencyAudit {
     };
   }
 
-  /**
-   * @param {{originalSize: number, webpSize: number}} image
-   * @return {{bytes: number, percent: number}}
-   */
-  static computeSavings(image) {
-    const bytes = image.originalSize - image.webpSize;
-    const percent = 100 * bytes / image.originalSize;
-    return {bytes, percent};
-  }
-
   /**
    * @param {{naturalWidth: number, naturalHeight: number}} imageElement
    * @return {number}
@@ -65,6 +55,38 @@ class ModernImageFormats extends ByteEfficiencyAudit {
     return Math.round(totalPixels * expectedBytesPerPixel);
   }
 
+  /**
+   * @param {{naturalWidth: number, naturalHeight: number}} imageElement
+   * @return {number}
+   */
+  static estimateAvifSizeFromDimensions(imageElement) {
+    const totalPixels = imageElement.naturalWidth * imageElement.naturalHeight;
+    // See above for the rationale behind our 2 byte-per-pixel baseline and WebP ratio of 10:1.
+    // AVIF usually gives ~20% additional savings on top of that, so we will use 12:1.
+    // This is quite pessimistic as Netflix study shows a photographic compression ratio of ~40:1
+    // (0.4 *bits* per pixel at SSIM 0.97).
+    // https://netflixtechblog.com/avif-for-next-generation-image-coding-b1d75675fe4
+    const expectedBytesPerPixel = 2 * 1 / 12;
+    return Math.round(totalPixels * expectedBytesPerPixel);
+  }
+
+  /**
+   * @param {{jpegSize: number | undefined, webpSize: number | undefined}} otherFormatSizes
+   * @return {number|undefined}
+   */
+  static estimateAvifSizeFromWebPAndJpegEstimates(otherFormatSizes) {
+    if (!otherFormatSizes.jpegSize || !otherFormatSizes.webpSize) return undefined;
+
+    // AVIF saves at least ~50% on JPEG, ~20% on WebP at low quality.
+    // http://downloads.aomedia.org/assets/pdf/symposium-2019/slides/CyrilConcolato_Netflix-AVIF-AOM-Research-Symposium-2019.pdf
+    // https://jakearchibald.com/2020/avif-has-landed/
+    // https://www.finally.agency/blog/what-is-avif-image-format
+    // See https://github.com/GoogleChrome/lighthouse/issues/12295#issue-840261460 for more.
+    const estimateFromJpeg = otherFormatSizes.jpegSize * 5 / 10;
+    const estimateFromWebp = otherFormatSizes.webpSize * 8 / 10;
+    return estimateFromJpeg / 2 + estimateFromWebp / 2;
+  }
+
   /**
    * @param {LH.Artifacts} artifacts
    * @return {ByteEfficiencyAudit.ByteEfficiencyProduct}
@@ -86,7 +108,15 @@ class ModernImageFormats extends ByteEfficiencyAudit {
         continue;
       }
 
+      // Skip if the image was already using a modern format.
+      if (image.mimeType === 'image/webp' || image.mimeType === 'image/avif') continue;
+
+      const jpegSize = image.jpegSize;
       let webpSize = image.webpSize;
+      let avifSize = ModernImageFormats.estimateAvifSizeFromWebPAndJpegEstimates({
+        jpegSize,
+        webpSize,
+      });
       let fromProtocol = true;
 
       if (typeof webpSize === 'undefined') {
@@ -107,21 +137,30 @@ class ModernImageFormats extends ByteEfficiencyAudit {
           naturalHeight,
           naturalWidth,
         });
+        avifSize = ModernImageFormats.estimateAvifSizeFromDimensions({
+          naturalHeight,
+          naturalWidth,
+        });
         fromProtocol = false;
       }
 
-      if (image.originalSize < webpSize + IGNORE_THRESHOLD_IN_BYTES) continue;
+      if (webpSize === undefined || avifSize === undefined) continue;
+
+      // Visible wasted bytes uses AVIF, but we still include the WebP savings in the LHR.
+      const wastedWebpBytes = image.originalSize - webpSize;
+      const wastedBytes = image.originalSize - avifSize;
+      if (wastedBytes < IGNORE_THRESHOLD_IN_BYTES) continue;
 
       const url = URL.elideDataURI(image.url);
       const isCrossOrigin = !URL.originsMatch(pageURL, image.url);
-      const webpSavings = ModernImageFormats.computeSavings({...image, webpSize: webpSize});
 
       items.push({
         url,
         fromProtocol,
         isCrossOrigin,
         totalBytes: image.originalSize,
-        wastedBytes: webpSavings.bytes,
+        wastedBytes,
+        wastedWebpBytes,
       });
     }
 
diff --git a/lighthouse-core/lib/i18n/locales/en-US.json b/lighthouse-core/lib/i18n/locales/en-US.json
index 95b1c960f..d3b91964c 100644
--- a/lighthouse-core/lib/i18n/locales/en-US.json
+++ b/lighthouse-core/lib/i18n/locales/en-US.json
@@ -477,7 +477,7 @@
     "message": "Avoid serving legacy JavaScript to modern browsers"
   },
   "lighthouse-core/audits/byte-efficiency/modern-image-formats.js | description": {
-    "message": "Image formats like JPEG 2000, JPEG XR, and WebP often provide better compression than PNG or JPEG, which means faster downloads and less data consumption. [Learn more](https://web.dev/uses-webp-images/)."
+    "message": "Image formats like WebP and AVIF often provide better compression than PNG or JPEG, which means faster downloads and less data consumption. [Learn more](https://web.dev/uses-webp-images/)."
   },
   "lighthouse-core/audits/byte-efficiency/modern-image-formats.js | title": {
     "message": "Serve images in next-gen formats"
diff --git a/lighthouse-core/lib/i18n/locales/en-XL.json b/lighthouse-core/lib/i18n/locales/en-XL.json
index 2c301c4be..d068a57c3 100644
--- a/lighthouse-core/lib/i18n/locales/en-XL.json
+++ b/lighthouse-core/lib/i18n/locales/en-XL.json
@@ -477,7 +477,7 @@
     "message": "Âv́ôíd̂ śêŕv̂ín̂ǵ l̂éĝáĉý Ĵáv̂áŜćr̂íp̂t́ t̂ó m̂ód̂ér̂ń b̂ŕôẃŝér̂ś"
   },
   "lighthouse-core/audits/byte-efficiency/modern-image-formats.js | description": {
-    "message": "Îḿâǵê f́ôŕm̂át̂ś l̂ík̂é ĴṔÊǴ 2000, ĴṔÊǴ X̂Ŕ, âńd̂ Ẃêb́P̂ óf̂t́êń p̂ŕôv́îd́ê b́êt́t̂ér̂ ćôḿp̂ŕêśŝíôń t̂h́âń P̂ŃĜ ór̂ J́P̂ÉĜ, ẃĥíĉh́ m̂éâńŝ f́âśt̂ér̂ d́ôẃn̂ĺôád̂ś âńd̂ ĺêśŝ d́ât́â ćôńŝúm̂ṕt̂íôń. [L̂éâŕn̂ ḿôŕê](https://web.dev/uses-webp-images/)."
+    "message": "Îḿâǵê f́ôŕm̂át̂ś l̂ík̂é Ŵéb̂Ṕ âńd̂ ÁV̂ÍF̂ óf̂t́êń p̂ŕôv́îd́ê b́êt́t̂ér̂ ćôḿp̂ŕêśŝíôń t̂h́âń P̂ŃĜ ór̂ J́P̂ÉĜ, ẃĥíĉh́ m̂éâńŝ f́âśt̂ér̂ d́ôẃn̂ĺôád̂ś âńd̂ ĺêśŝ d́ât́â ćôńŝúm̂ṕt̂íôń. [L̂éâŕn̂ ḿôŕê](https://web.dev/uses-webp-images/)."
   },
   "lighthouse-core/audits/byte-efficiency/modern-image-formats.js | title": {
     "message": "Ŝér̂v́ê ím̂áĝéŝ ín̂ ńêx́t̂-ǵêń f̂ór̂ḿât́ŝ"


Test patch:
diff --git a/lighthouse-cli/test/smokehouse/test-definitions/byte-efficiency/expectations.js b/lighthouse-cli/test/smokehouse/test-definitions/byte-efficiency/expectations.js
index b9f4a8f6d..141492041 100644
--- a/lighthouse-cli/test/smokehouse/test-definitions/byte-efficiency/expectations.js
+++ b/lighthouse-cli/test/smokehouse/test-definitions/byte-efficiency/expectations.js
@@ -215,10 +215,15 @@ const expectations = [
         },
         'modern-image-formats': {
           details: {
-            overallSavingsBytes: '>60000',
-            items: {
-              length: 6,
-            },
+            overallSavingsBytes: '137000 +/- 10000',
+            items: [
+              {url: /lighthouse-1024x680.jpg$/},
+              {url: /lighthouse-unoptimized.jpg$/},
+              {url: /lighthouse-480x320.jpg$/},
+              {url: /lighthouse-480x320.jpg\?attributesized/},
+              {url: /lighthouse-480x320.jpg\?css/},
+              {url: /lighthouse-480x320.jpg\?sprite/},
+            ],
           },
         },
         'uses-text-compression': {
diff --git a/lighthouse-core/test/audits/byte-efficiency/modern-image-formats-test.js b/lighthouse-core/test/audits/byte-efficiency/modern-image-formats-test.js
index dfc8ce0b8..5d82f9d2c 100644
--- a/lighthouse-core/test/audits/byte-efficiency/modern-image-formats-test.js
+++ b/lighthouse-core/test/audits/byte-efficiency/modern-image-formats-test.js
@@ -48,14 +48,14 @@ function generateArtifacts(images) {
 
 describe('Page uses optimized images', () => {
   it('ignores files when there is only insignificant savings', () => {
-    const artifacts = generateArtifacts([{originalSize: 5000, webpSize: 4500}]);
+    const artifacts = generateArtifacts([{originalSize: 5000, jpegSize: 10000, webpSize: 4500}]);
     const auditResult = ModernImageFormats.audit_(artifacts);
 
     expect(auditResult.items).toEqual([]);
   });
 
-  it('flags files when there is only small savings', () => {
-    const artifacts = generateArtifacts([{originalSize: 15000, webpSize: 4500}]);
+  it('flags files using AVIF savings', () => {
+    const artifacts = generateArtifacts([{originalSize: 15000, jpegSize: 8000, webpSize: 5000}]);
     const auditResult = ModernImageFormats.audit_(artifacts);
 
     expect(auditResult.items).toEqual([
@@ -63,7 +63,8 @@ describe('Page uses optimized images', () => {
         fromProtocol: true,
         isCrossOrigin: false,
         totalBytes: 15000,
-        wastedBytes: 15000 - 4500,
+        wastedBytes: 15000 - (5000 * 0.8),
+        wastedWebpBytes: 15000 - 5000,
         url: 'http://google.com/image.jpeg',
       },
     ]);
@@ -78,7 +79,8 @@ describe('Page uses optimized images', () => {
         fromProtocol: false,
         isCrossOrigin: false,
         totalBytes: 1e6,
-        wastedBytes: 1e6 - 1000 * 1000 * 2 / 10,
+        wastedBytes: Math.round(1e6 - 1000 * 1000 * 2 / 12),
+        wastedWebpBytes: Math.round(1e6 - 1000 * 1000 * 2 / 10),
         url: 'http://google.com/image.jpeg',
       },
     ]);
@@ -86,7 +88,7 @@ describe('Page uses optimized images', () => {
 
   it('estimates savings on cross-origin files', () => {
     const artifacts = generateArtifacts([{
-      url: 'http://localhost:1234/image.jpeg', originalSize: 50000, webpSize: 20000,
+      url: 'http://localhost:1234/image.jpg', originalSize: 50000, jpegSize: 50000, webpSize: 20000,
     }]);
     const auditResult = ModernImageFormats.audit_(artifacts);
 
@@ -94,16 +96,27 @@ describe('Page uses optimized images', () => {
       {
         fromProtocol: true,
         isCrossOrigin: true,
-        url: 'http://localhost:1234/image.jpeg',
+        url: 'http://localhost:1234/image.jpg',
       },
     ]);
   });
 
   it('passes when all images are sufficiently optimized', () => {
     const artifacts = generateArtifacts([
-      {type: 'png', originalSize: 50000, webpSize: 50001},
-      {type: 'jpeg', originalSize: 50000, webpSize: 50001},
-      {type: 'bmp', originalSize: 4000, webpSize: 2000},
+      {type: 'png', originalSize: 50000, jpegSize: 100001, webpSize: 60001},
+      {type: 'jpeg', originalSize: 50000, jpegSize: 100001, webpSize: 60001},
+      {type: 'bmp', originalSize: 4000, webpSize: 5001},
+    ]);
+
+    const auditResult = ModernImageFormats.audit_(artifacts);
+
+    expect(auditResult.items).toEqual([]);
+  });
+
+  it('passes when all images are already using a modern format', () => {
+    const artifacts = generateArtifacts([
+      {type: 'webp', originalSize: 50000, jpegSize: 100000, webpSize: 100000},
+      {type: 'avif', originalSize: 50000, jpegSize: 100000, webpSize: 100000},
     ]);
 
     const auditResult = ModernImageFormats.audit_(artifacts);
@@ -113,7 +126,7 @@ describe('Page uses optimized images', () => {
 
   it('elides data URIs', () => {
     const artifacts = generateArtifacts([
-      {type: 'data:webp', originalSize: 15000, webpSize: 4500},
+      {type: 'data:jpeg', originalSize: 15000, jpegSize: 10000, webpSize: 4500},
     ]);
 
     const auditResult = ModernImageFormats.audit_(artifacts);
diff --git a/lighthouse-core/test/results/sample_v2.json b/lighthouse-core/test/results/sample_v2.json
index d22c433a1..8b14cdf2f 100644
--- a/lighthouse-core/test/results/sample_v2.json
+++ b/lighthouse-core/test/results/sample_v2.json
@@ -3823,12 +3823,12 @@
     "modern-image-formats": {
       "id": "modern-image-formats",
       "title": "Serve images in next-gen formats",
-      "description": "Image formats like JPEG 2000, JPEG XR, and WebP often provide better compression than PNG or JPEG, which means faster downloads and less data consumption. [Learn more](https://web.dev/uses-webp-images/).",
+      "description": "Image formats like WebP and AVIF often provide better compression than PNG or JPEG, which means faster downloads and less data consumption. [Learn more](https://web.dev/uses-webp-images/).",
       "score": 0.88,
       "scoreDisplayMode": "numeric",
       "numericValue": 150,
       "numericUnit": "millisecond",
-      "displayValue": "Potential savings of 9 KiB",
+      "displayValue": "Potential savings of 13 KiB",
       "warnings": [],
       "details": {
         "type": "opportunity",
@@ -3860,11 +3860,12 @@
             "fromProtocol": true,
             "isCrossOrigin": false,
             "totalBytes": 24620,
-            "wastedBytes": 9028
+            "wastedBytes": 12989.45,
+            "wastedWebpBytes": 9028
           }
         ],
         "overallSavingsMs": 150,
-        "overallSavingsBytes": 9028
+        "overallSavingsBytes": 12989.45
       }
     },
     "uses-optimized-images": {
@@ -8172,7 +8173,7 @@
         },
         {
           "values": {
-            "wastedBytes": 9028
+            "wastedBytes": 12989.45
           },
           "path": "audits[modern-image-formats].displayValue"
         },
diff --git a/report/test/renderer/report-ui-features-test.js b/report/test/renderer/report-ui-features-test.js
index 8db7fb90b..941503e83 100644
--- a/report/test/renderer/report-ui-features-test.js
+++ b/report/test/renderer/report-ui-features-test.js
@@ -129,7 +129,10 @@ describe('ReportUIFeatures', () => {
       beforeAll(() => {
         const lhr = JSON.parse(JSON.stringify(sampleResults));
         lhr.requestedUrl = lhr.finalUrl = 'http://www.example.com';
-        const webpAuditItemTemplate = sampleResults.audits['modern-image-formats'].details.items[0];
+        const webpAuditItemTemplate = {
+          ...sampleResults.audits['modern-image-formats'].details.items[0],
+          wastedBytes: 8.8 * 1024,
+        };
         const renderBlockingAuditItemTemplate =
           sampleResults.audits['render-blocking-resources'].details.items[0];
         const textCompressionAuditItemTemplate =