Repository: gpuweb/cts
Pull request #4365: cameraCapture: Simplify codepaths, try to wait for a real frame
Author: login: kainino0x
Created: 2025-04-19T03:14:08Z | Merged: 2025-04-23T06:41:10Z

- Don't use fallback paths in camera capture; instead, have the test pick the exact path it's using, and parameterize over it. Test two paths: VideoFrame from MediaStreamTrackProcessor, and the old-fashioned HTMLVideoElement.
- Work around an issue where Chrome (at least on Mac) shows blank frames for a while after initializing the camera.
- I happened to notice that requesting different width/height from the camera is broken in Chrome in several ways, so added cases for that.
- Added a TODO for copyExternalImageToTexture from camera

These tests only worked in Chrome, and even then they were buggy (usually running the test on a blank first frame). Now, the HTMLVideoElement tests work in Safari.

Test failures are as follows, on an M1 Mac:
- Chrome: https://crbug.com/411656657
  - HTMLVideoElement passes some cases, but fails when a requested size is passed to getUserMedia().
  - VideoFrame fails due to incorrect color management (regardless of dstColorSpace).
- Safari:
  - HTMLVideoElement fails due to incorrect color management (regardless of dstColorSpace).
  - VideoFrame is skipped.
- Firefox: doesn't yet implement importExternalTexture.

Issue: fixes #4363, cc #4364

<hr>

**Requirements for PR author:**

- [x] All missing test coverage is tracked with "TODO" or `.unimplemented()`.
- [x] New helpers are `/** documented */` and new helper files are found in `helper_index.txt`.
- [x] Test behaves as expected in a WebGPU implementation. (If not passing, explain above.)
- [x] Test have be tested with compatibility mode validation enabled and behave as expected. (If not passing, explain above.)

**Requirements for [reviewer sign-off](https://github.com/gpuweb/cts/blob/main/docs/reviews.md):**

- [x] Tests are properly located in the test tree.
- [x] [Test descriptions](https://github.com/gpuweb/cts/blob/main/docs/intro/plans.md) allow a reader to "read only the test plans and evaluate coverage completeness", and accurately reflect the test code.
- [x] Tests provide complete coverage (including validation control cases). **Missing coverage MUST be covered by TODOs.**
- [x] Helpers and types promote readability and maintainability.

When landing this PR, be sure to make any necessary issue status updates.


Linked issue #4363: cameraCapture test doesn't reliably get a real frame
The cameraCapture test from #3874 doesn't reliably get a real video frame. In Chrome on Mac, I'm seeing that the first time I run the test, it passes, but then if I run it again, it fails:
<https://gpuweb.github.io/cts/standalone/?q=webgpu:web_platform,external_texture,video:importExternalTexture,cameraCapture:dstColorSpace=%22srgb%22>

If I display the image with:
```
    document.body.append(canvas);
```

it is black on the first run, and has a real image on the second run. When it has a real image, it fails, usually with a margin of around 0.05. I don't know why.

I found this because if I use `--use-fake-device-for-media-stream --auto-accept-camera-and-microphone-capture` in Chrome to test with a fake camera, then it gets a real frame on the first try, and that causes it to fail.

cc @shaoboyan
Linked issue comments: user: login: kainino0x
created_at: 2025-04-19T01:13:12Z
body: Another comment about the code that gets video frames:

There are three codepaths. Chrome always takes the first path, the other paths don't seem to work, even in Chrome (the ImageCapture path throws an exception, the video element path. We should not have fallbacks, we should pick the path deterministically (use one path for the main thread, which should never skip, and another path for worker threads, which may skip).

Safari: `EXCEPTION: TypeError: imageCapture.grabFrame is not a function. (In 'imageCapture.grabFrame()', 'imageCapture.grabFrame' is undefined)`
Firefox: `SKIP: HTMLVideoElement.captureStream is not supported`
author_association: COLLABORATOR
user: login: shaoboyan091
created_at: 2025-04-21T01:24:38Z
body: > The cameraCapture test from [#3874](https://github.com/gpuweb/cts/pull/3874) doesn't reliably get a real video frame. In Chrome on Mac, I'm seeing that the first time I run the test, it passes, but then if I run it again, it fails: https://gpuweb.github.io/cts/standalone/?q=webgpu:web_platform,external_texture,video:importExternalTexture,cameraCapture:dstColorSpace=%22srgb%22
> 
> If I display the image with:
> 
> ```
>     document.body.append(canvas);
> ```
> 
> it is black on the first run, and has a real image on the second run. When it has a real image, it fails, usually with a margin of around 0.05. I don't know why.
> 
> I found this because if I use `--use-fake-device-for-media-stream --auto-accept-camera-and-microphone-capture` in Chrome to test with a fake camera, then it gets a real frame on the first try, and that causes it to fail.
> 
> cc [@shaoboyan](https://github.com/shaoboyan)

Thanks for ccing. FYI, my previous github account @shaoboyan  has lost due to I linked all my recover methods to intel company email address and forget to change that... Now I used @shaoboyan091  instead. I see your fix PR, is there anything need to be further investigated? (e.g. https://github.com/gpuweb/cts/issues/4364)?
author_association: CONTRIBUTOR
user: login: kainino0x
created_at: 2025-04-21T23:45:36Z
body: > FYI, my previous github account [@shaoboyan](https://github.com/shaoboyan) has lost due to I linked all my recover methods to intel company email address and forget to change that... Now I used [@shaoboyan091](https://github.com/shaoboyan091) instead.

Oops, I forgot. I've removed the old account from the gpuweb org, that should make it easier to remember.

> I see your fix PR, is there anything need to be further investigated? (e.g. [#4364](https://github.com/gpuweb/cts/issues/4364))?

Eventually yes, we should add some more coverage here, but let's get a fix landed first.
author_association: COLLABORATOR

Diff:
diff --git a/src/webgpu/web_platform/external_texture/video.spec.ts b/src/webgpu/web_platform/external_texture/video.spec.ts
index 35cca1703084..e23a56680f2d 100644
--- a/src/webgpu/web_platform/external_texture/video.spec.ts
+++ b/src/webgpu/web_platform/external_texture/video.spec.ts
@@ -6,9 +6,11 @@ Tests for external textures from HTMLVideoElement (and other video-type sources?
 
 TODO: consider whether external_texture and copyToTexture video tests should be in the same file
 TODO(#3193): Test video in BT.2020 color space
+TODO(#4364): Test camera capture with copyExternalImageToTexture (not necessarily in this file)
 `;
 
 import { makeTestGroup } from '../../../common/framework/test_group.js';
+import { unreachable } from '../../../common/util/util.js';
 import { GPUTest } from '../../gpu_test.js';
 import * as ttu from '../../texture_test_utils.js';
 import { TextureUploadingUtils } from '../../util/copy_to_texture.js';
@@ -17,12 +19,13 @@ import {
   startPlayingAndWaitForVideo,
   getVideoFrameFromVideoElement,
   getVideoElement,
-  captureCameraFrame,
   convertToUnorm8,
   kPredefinedColorSpace,
   kVideoNames,
   kVideoInfo,
   kVideoExpectedColors,
+  getVideoElementFromCamera,
+  getVideoFrameFromCamera,
 } from '../../web_platform/util.js';
 
 const kHeight = 16;
@@ -633,34 +636,64 @@ compared with 2d canvas rendering result.
 `
   )
   .params(u =>
+    // NOTE: Be careful not to add too many parameters here, this test is SLOW to initialize!
     u //
       .combineWithParams(checkNonStandardIsZeroCopyIfAvailable())
+      .combine('path', ['HTMLVideoElement', 'MediaStreamTrackProcessor'] as const)
       .combine('dstColorSpace', kPredefinedColorSpace)
+      .combine('constraints', [
+        true,
+        { width: 64, height: 48 },
+        { width: 100, height: 300 },
+      ] as const)
   )
   .fn(async t => {
-    const { dstColorSpace } = t.params;
-
-    const frame = await captureCameraFrame(t);
-
-    if (frame.displayHeight === 0 || frame.displayWidth === 0) {
-      t.skip('Captured video frame has 0 height or width.');
+    const { path, dstColorSpace, constraints } = t.params;
+
+    // Enable this while debugging to show the "actual" and "expected" canvases on screen.
+    const kDebugShowCanvasesOnScreen = false;
+
+    let source: HTMLVideoElement | VideoFrame;
+    let frameWidth: number, frameHeight: number;
+    switch (path) {
+      case 'HTMLVideoElement': {
+        const video = await getVideoElementFromCamera(t, constraints, true);
+        frameWidth = video.videoWidth;
+        frameHeight = video.videoHeight;
+        source = video;
+        break;
+      }
+      case 'MediaStreamTrackProcessor': {
+        const frame = await getVideoFrameFromCamera(t, constraints);
+        frameWidth = frame.displayWidth;
+        frameHeight = frame.displayHeight;
+        source = frame;
+        break;
+      }
+      default:
+        unreachable();
     }
 
-    const frameWidth = frame.displayWidth;
-    const frameHeight = frame.displayHeight;
-
     // Use WebGPU + GPUExternalTexture to render the captured frame.
-    const colorAttachment = t.createTextureTracked({
+    const webgpuCanvas = createCanvas(t, 'onscreen', frameWidth, frameHeight);
+    if (kDebugShowCanvasesOnScreen) {
+      document.body.append(document.createElement('br'));
+      document.body.append(webgpuCanvas);
+    }
+
+    const webgpuContext = webgpuCanvas.getContext('webgpu')!;
+    webgpuContext.configure({
+      device: t.device,
       format: kFormat,
-      size: { width: frameWidth, height: frameHeight },
-      usage: GPUTextureUsage.COPY_SRC | GPUTextureUsage.RENDER_ATTACHMENT,
+      usage: GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.COPY_SRC,
     });
+    const colorAttachment = webgpuContext.getCurrentTexture();
 
     const pipeline = createExternalTextureSamplingTestPipeline(t);
     const bindGroup = createExternalTextureSamplingTestBindGroup(
       t,
       t.params.checkNonStandardIsZeroCopy,
-      frame,
+      source,
       pipeline,
       dstColorSpace
     );
@@ -684,6 +717,9 @@ compared with 2d canvas rendering result.
 
     // Use 2d context canvas as expected result.
     const canvas = createCanvas(t, 'onscreen', frameWidth, frameHeight);
+    if (kDebugShowCanvasesOnScreen) {
+      document.body.append(canvas);
+    }
 
     const canvasContext = canvas.getContext('2d', { colorSpace: dstColorSpace });
 
@@ -692,7 +728,7 @@ compared with 2d canvas rendering result.
     }
 
     const ctx = canvasContext as CanvasRenderingContext2D;
-    ctx.drawImage(frame, 0, 0, frameWidth, frameHeight);
+    ctx.drawImage(source, 0, 0, frameWidth, frameHeight);
 
     const imageData = ctx.getImageData(0, 0, frameWidth, frameHeight, {
       colorSpace: dstColorSpace,
diff --git a/src/webgpu/web_platform/util.ts b/src/webgpu/web_platform/util.ts
index cc6f3c809b83..dcb0d25b72cf 100644
--- a/src/webgpu/web_platform/util.ts
+++ b/src/webgpu/web_platform/util.ts
@@ -2,7 +2,7 @@ import { Fixture, SkipTestCase } from '../../common/framework/fixture.js';
 import { getResourcePath } from '../../common/framework/resources.js';
 import { keysOf } from '../../common/util/data_tables.js';
 import { timeout } from '../../common/util/timeout.js';
-import { ErrorWithExtra, raceWithRejectOnTimeout } from '../../common/util/util.js';
+import { ErrorWithExtra, assert, raceWithRejectOnTimeout } from '../../common/util/util.js';
 import { GPUTest } from '../gpu_test.js';
 import { RGBA, srgbToDisplayP3 } from '../util/color_space_conversion.js';
 
@@ -559,14 +559,12 @@ function callbackHelper(
 }
 
 /**
- * Create VideoFrame from camera captured frame. Check whether browser environment has
- * camera supported.
- * Returns a webcodec VideoFrame.
- *
- * @param test: GPUTest that requires getting VideoFrame
- *
+ * Get a MediaStream from the default webcam via `getUserMedia()`.
  */
-export async function captureCameraFrame(test: GPUTest): Promise<VideoFrame> {
+async function getStreamFromCamera(
+  test: Fixture,
+  videoTrackConstraints: MediaTrackConstraints | true
+): Promise<MediaStream> {
   test.skipIf(typeof navigator === 'undefined', 'navigator does not exist in this environment');
   test.skipIf(
     typeof navigator.mediaDevices === 'undefined' ||
@@ -574,44 +572,132 @@ export async function captureCameraFrame(test: GPUTest): Promise<VideoFrame> {
     "Browser doesn't support capture frame from camera."
   );
 
-  const stream = await navigator.mediaDevices.getUserMedia({ video: true });
-  const track = stream.getVideoTracks()[0] as MediaStreamVideoTrack;
-
-  test.skipIf(!track, "Doesn't have valid camera captured stream for testing.");
+  const stream = await navigator.mediaDevices.getUserMedia({
+    audio: false,
+    video: videoTrackConstraints,
+  });
+  test.trackForCleanup({
+    close() {
+      for (const track of stream.getTracks()) {
+        track.stop();
+      }
+    },
+  });
+  return stream;
+}
 
-  // Use MediaStreamTrackProcessor and ReadableStream to generate video frame directly.
-  if (typeof MediaStreamTrackProcessor !== 'undefined') {
-    const trackProcessor = new MediaStreamTrackProcessor({ track });
-    const reader = trackProcessor.readable.getReader();
-    const result = await reader.read();
-    if (result.done) {
-      test.skip('MediaStreamTrackProcessor: Cannot get valid frame from readable stream.');
+/**
+ * Chrome on macOS (at least) takes a while before it switches from blank frames
+ * to real frames. Wait up to 50 frames for something to show up on the camera.
+ */
+async function waitForNonBlankFrame({
+  getSource,
+  waitForNextFrame,
+}: {
+  getSource: () => HTMLVideoElement | VideoFrame;
+  waitForNextFrame: () => Promise<void>;
+}) {
+  const cvs = document.createElement('canvas');
+  [cvs.width, cvs.height] = [4, 4];
+  const ctx = cvs.getContext('2d', { willReadFrequently: true })!;
+  let foundNonBlankFrame = false;
+  for (let i = 0; i < 50; ++i) {
+    ctx.drawImage(getSource(), 0, 0, cvs.width, cvs.height);
+    const pixels = new Uint32Array(ctx.getImageData(0, 0, cvs.width, cvs.height).data.buffer);
+    // Look only at RGB, ignore alpha.
+    if (pixels.some(p => (p & 0x00ffffff) !== 0)) {
+      foundNonBlankFrame = true;
+      break;
     }
-
-    return result.value;
-  }
-
-  // Fallback to ImageCapture if MediaStreamTrackProcessor not supported. Using grabFrame() to
-  // generate imageBitmap and creating video frame from it.
-  if (typeof ImageCapture !== 'undefined') {
-    const imageCapture = new ImageCapture(track);
-    const imageBitmap = await imageCapture.grabFrame();
-    return new VideoFrame(imageBitmap);
+    await waitForNextFrame();
   }
+  assert(foundNonBlankFrame, 'Failed to get a non-blank video frame');
+}
 
-  // Fallback to using HTMLVideoElement to do capture.
+/**
+ * Uses MediaStreamTrackProcessor to capture a VideoFrame from the camera.
+ * Skips the test if not supported.
+ * @param videoTrackConstraints - MediaTrackConstraints (e.g. width/height) to pass to
+ *     `getUserMedia()`, or `true` if none.
+ */
+export async function getVideoFrameFromCamera(
+  test: Fixture,
+  videoTrackConstraints: MediaTrackConstraints | true
+): Promise<VideoFrame> {
   test.skipIf(
-    typeof HTMLVideoElement === 'undefined',
-    'Try to use HTMLVideoElement do capture but HTMLVideoElement not available.'
+    typeof MediaStreamTrackProcessor === 'undefined',
+    'MediaStreamTrackProcessor not supported'
   );
 
+  const stream = await getStreamFromCamera(test, videoTrackConstraints);
+  const tracks = stream.getVideoTracks();
+  assert(tracks.length > 0, 'no tracks found');
+  const track = tracks[0] as MediaStreamVideoTrack;
+
+  const trackProcessor = new MediaStreamTrackProcessor({ track });
+  const reader = trackProcessor.readable.getReader();
+
+  const waitForNextFrame = async () => {
+    const result = await reader.read();
+    assert(!result.done, "MediaStreamTrackProcessor: Couldn't get valid frame from stream.");
+    return result.value;
+  };
+  let frame: VideoFrame = await waitForNextFrame();
+  await waitForNonBlankFrame({
+    getSource: () => frame,
+    async waitForNextFrame() {
+      frame.close();
+      frame = await waitForNextFrame();
+    },
+  });
+
+  test.trackForCleanup(frame);
+  return frame;
+}
+
+/**
+ * Create an HTMLVideoElement from the camera stream. Skips the test if not supported.
+ * @param videoTrackConstraints - MediaTrackConstraints (e.g. width/height) to pass to
+ *     `getUserMedia()`, or `true` if none.
+ * @param paused - whether the video should be paused before returning.
+ */
+export async function getVideoElementFromCamera(
+  test: Fixture,
+  videoTrackConstraints: MediaTrackConstraints | true,
+  paused: boolean
+): Promise<HTMLVideoElement> {
+  const stream = await getStreamFromCamera(test, videoTrackConstraints);
+
+  // Main thread
   const video = document.createElement('video');
+  video.loop = false;
+  video.muted = true;
+  video.setAttribute('playsinline', '');
   video.srcObject = stream;
+  await new Promise(resolve => {
+    video.onloadedmetadata = resolve;
+  });
+  await startPlayingAndWaitForVideo(video, () => {});
 
-  const frame = await getVideoFrameFromVideoElement(test, video);
-  test.trackForCleanup(frame);
+  await waitForNonBlankFrame({
+    getSource: () => video,
+    waitForNextFrame: () =>
+      new Promise(resolve =>
+        video.requestVideoFrameCallback(() => {
+          resolve();
+        })
+      ),
+  });
 
-  return frame;
+  if (paused) {
+    // Pause the video so we get consistent readbacks.
+    await new Promise(resolve => {
+      video.onpause = resolve;
+      video.pause();
+    });
+  }
+
+  return video;
 }
 
 const kFourColorsInfo = {


Reviews:
user: login: shaoboyan091
body: Thanks for fixing! Some questions.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T02:47:13Z
user: login: shaoboyan091
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T02:49:30Z
user: login: kainino0x
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-21T23:48:00Z
user: login: kainino0x
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-21T23:48:43Z
user: login: shaoboyan091
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T01:00:29Z
user: login: kainino0x
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-22T02:12:55Z
user: login: kainino0x
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-23T02:43:59Z
user: login: kainino0x
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-23T03:14:55Z
user: login: kainino0x
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-23T03:16:06Z
user: login: kainino0x
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-23T03:16:44Z
user: login: kainino0x
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-23T03:37:51Z
user: login: kainino0x
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-23T05:10:16Z
user: login: shaoboyan091
body: Looks great! 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-23T06:32:46Z

Review comments:
diff_hunk: @@ -633,34 +636,63 @@ compared with 2d canvas rendering result.
 `
   )
   .params(u =>
+    // NOTE: Be careful not to add too many parameters here, this test is SLOW to initialize!
     u //
       .combineWithParams(checkNonStandardIsZeroCopyIfAvailable())
+      .combine('path', ['HTMLVideoElement', 'MediaStreamTrackProcessor'] as const)
       .combine('dstColorSpace', kPredefinedColorSpace)
+      .combine('requestSize', [undefined, [64, 48], [100, 300]] as const)
   )
   .fn(async t => {
-    const { dstColorSpace } = t.params;
-
-    const frame = await captureCameraFrame(t);
-
-    if (frame.displayHeight === 0 || frame.displayWidth === 0) {
-      t.skip('Captured video frame has 0 height or width.');
+    const { path, dstColorSpace, requestSize } = t.params;
+
+    // Enable this while debugging to show the "actual" and "expected" canvases on screen.
+    const kDebugShowCanvasesOnScreen = false;
+
+    let source: HTMLVideoElement | VideoFrame;
+    let frameWidth: number, frameHeight: number;
+    switch (path) {
+      case 'HTMLVideoElement': {
+        const video = await getVideoElementFromCamera(t, requestSize?.[0], requestSize?.[1]);
+        // Pause the video so we get consistent readbacks. Wait a bit to be sure.
+        video.pause();
+        await resolveOnTimeout(10);
path: src/webgpu/web_platform/external_texture/video.spec.ts
user: login: shaoboyan091
body: Would rvfc (Or another startPlayingAndWaitForVideo) helps? Just curious, why 10ms? Is it from experiments or some implementation details in chrome?
created_at: 2025-04-21T01:34:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -692,12 +727,22 @@ compared with 2d canvas rendering result.
     }
 
     const ctx = canvasContext as CanvasRenderingContext2D;
-    ctx.drawImage(frame, 0, 0, frameWidth, frameHeight);
+    ctx.drawImage(source, 0, 0, frameWidth, frameHeight);
 
     const imageData = ctx.getImageData(0, 0, frameWidth, frameHeight, {
       colorSpace: dstColorSpace,
     });
 
+    // Make sure the expected frame is not all black.
+    let isAllBlack = true;
+    for (let i = 0; i < imageData.data.length; i += 4) {
+      if (imageData.data[i] > 2 || imageData.data[i + 1] > 2 || imageData.data[i + 2] > 2) {
+        isAllBlack = false;
+        break;
+      }
+    }
+    t.expect(!isAllBlack, 'Video frame should not be all black');
path: src/webgpu/web_platform/external_texture/video.spec.ts
user: login: shaoboyan091
body: If we captured all black frames, shall we try to capture another frame until there is no all black frame?
created_at: 2025-04-21T01:40:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -558,60 +563,85 @@ function callbackHelper(
   return { promise, callbackAndResolve: callbackAndResolve! };
 }
 
-/**
- * Create VideoFrame from camera captured frame. Check whether browser environment has
- * camera supported.
- * Returns a webcodec VideoFrame.
- *
- * @param test: GPUTest that requires getting VideoFrame
- *
- */
-export async function captureCameraFrame(test: GPUTest): Promise<VideoFrame> {
+async function getStreamFromCamera(
+  test: Fixture,
+  width: number | undefined,
+  height: number | undefined
+): Promise<MediaStream> {
   test.skipIf(typeof navigator === 'undefined', 'navigator does not exist in this environment');
   test.skipIf(
     typeof navigator.mediaDevices === 'undefined' ||
       typeof navigator.mediaDevices.getUserMedia === 'undefined',
     "Browser doesn't support capture frame from camera."
   );
 
-  const stream = await navigator.mediaDevices.getUserMedia({ video: true });
-  const track = stream.getVideoTracks()[0] as MediaStreamVideoTrack;
+  const stream = await navigator.mediaDevices.getUserMedia({
+    audio: false,
+    video: { width, height },
+  });
+  test.trackForCleanup({
+    close() {
+      for (const track of stream.getTracks()) {
+        track.stop();
+      }
+    },
+  });
+  return stream;
+}
+
+/**
+ * Uses MediaStreamTrackProcessor to capture a VideoFrame from the camera.
+ * Skips the test if not supported.
+ */
path: src/webgpu/web_platform/util.ts
user: login: shaoboyan091
body: Missing @param for params description. (This is helper function.)
created_at: 2025-04-21T02:49:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -633,34 +636,63 @@ compared with 2d canvas rendering result.
 `
   )
   .params(u =>
+    // NOTE: Be careful not to add too many parameters here, this test is SLOW to initialize!
     u //
       .combineWithParams(checkNonStandardIsZeroCopyIfAvailable())
+      .combine('path', ['HTMLVideoElement', 'MediaStreamTrackProcessor'] as const)
       .combine('dstColorSpace', kPredefinedColorSpace)
+      .combine('requestSize', [undefined, [64, 48], [100, 300]] as const)
   )
   .fn(async t => {
-    const { dstColorSpace } = t.params;
-
-    const frame = await captureCameraFrame(t);
-
-    if (frame.displayHeight === 0 || frame.displayWidth === 0) {
-      t.skip('Captured video frame has 0 height or width.');
+    const { path, dstColorSpace, requestSize } = t.params;
+
+    // Enable this while debugging to show the "actual" and "expected" canvases on screen.
+    const kDebugShowCanvasesOnScreen = false;
+
+    let source: HTMLVideoElement | VideoFrame;
+    let frameWidth: number, frameHeight: number;
+    switch (path) {
+      case 'HTMLVideoElement': {
+        const video = await getVideoElementFromCamera(t, requestSize?.[0], requestSize?.[1]);
+        // Pause the video so we get consistent readbacks. Wait a bit to be sure.
+        video.pause();
+        await resolveOnTimeout(10);
path: src/webgpu/web_platform/external_texture/video.spec.ts
user: login: kainino0x
body: Just experimentally. I tried a number of things like getting multiple frames before starting, but none of them seemed to really work. I don't have any idea how long it's supposed to take, this just works on my machine. I think it's actually likely to be a property of the builtin camera driver on my Mac, not even an issue with Chrome.

Fortunately in Chrome testing, when we use a fake camera device, I don't see any initial blank frames (even on Mac), so this shouldn't be unreliable on testing bots.
created_at: 2025-04-21T23:47:59Z
author_association: COLLABORATOR
diff_hunk: @@ -692,12 +727,22 @@ compared with 2d canvas rendering result.
     }
 
     const ctx = canvasContext as CanvasRenderingContext2D;
-    ctx.drawImage(frame, 0, 0, frameWidth, frameHeight);
+    ctx.drawImage(source, 0, 0, frameWidth, frameHeight);
 
     const imageData = ctx.getImageData(0, 0, frameWidth, frameHeight, {
       colorSpace: dstColorSpace,
     });
 
+    // Make sure the expected frame is not all black.
+    let isAllBlack = true;
+    for (let i = 0; i < imageData.data.length; i += 4) {
+      if (imageData.data[i] > 2 || imageData.data[i + 1] > 2 || imageData.data[i + 2] > 2) {
+        isAllBlack = false;
+        break;
+      }
+    }
+    t.expect(!isAllBlack, 'Video frame should not be all black');
path: src/webgpu/web_platform/external_texture/video.spec.ts
user: login: kainino0x
body: Probably a good idea - with some long timeout (wouldn't want to loop forever because the camera is covered up or something)
created_at: 2025-04-21T23:48:43Z
author_association: COLLABORATOR
diff_hunk: @@ -633,34 +636,63 @@ compared with 2d canvas rendering result.
 `
   )
   .params(u =>
+    // NOTE: Be careful not to add too many parameters here, this test is SLOW to initialize!
     u //
       .combineWithParams(checkNonStandardIsZeroCopyIfAvailable())
+      .combine('path', ['HTMLVideoElement', 'MediaStreamTrackProcessor'] as const)
       .combine('dstColorSpace', kPredefinedColorSpace)
+      .combine('requestSize', [undefined, [64, 48], [100, 300]] as const)
   )
   .fn(async t => {
-    const { dstColorSpace } = t.params;
-
-    const frame = await captureCameraFrame(t);
-
-    if (frame.displayHeight === 0 || frame.displayWidth === 0) {
-      t.skip('Captured video frame has 0 height or width.');
+    const { path, dstColorSpace, requestSize } = t.params;
+
+    // Enable this while debugging to show the "actual" and "expected" canvases on screen.
+    const kDebugShowCanvasesOnScreen = false;
+
+    let source: HTMLVideoElement | VideoFrame;
+    let frameWidth: number, frameHeight: number;
+    switch (path) {
+      case 'HTMLVideoElement': {
+        const video = await getVideoElementFromCamera(t, requestSize?.[0], requestSize?.[1]);
+        // Pause the video so we get consistent readbacks. Wait a bit to be sure.
+        video.pause();
+        await resolveOnTimeout(10);
path: src/webgpu/web_platform/external_texture/video.spec.ts
user: login: shaoboyan091
body: Thanks! It's great that test bots don't have this blank frames issue.
created_at: 2025-04-22T01:00:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -633,34 +636,63 @@ compared with 2d canvas rendering result.
 `
   )
   .params(u =>
+    // NOTE: Be careful not to add too many parameters here, this test is SLOW to initialize!
     u //
       .combineWithParams(checkNonStandardIsZeroCopyIfAvailable())
+      .combine('path', ['HTMLVideoElement', 'MediaStreamTrackProcessor'] as const)
       .combine('dstColorSpace', kPredefinedColorSpace)
+      .combine('requestSize', [undefined, [64, 48], [100, 300]] as const)
   )
   .fn(async t => {
-    const { dstColorSpace } = t.params;
-
-    const frame = await captureCameraFrame(t);
-
-    if (frame.displayHeight === 0 || frame.displayWidth === 0) {
-      t.skip('Captured video frame has 0 height or width.');
+    const { path, dstColorSpace, requestSize } = t.params;
+
+    // Enable this while debugging to show the "actual" and "expected" canvases on screen.
+    const kDebugShowCanvasesOnScreen = false;
+
+    let source: HTMLVideoElement | VideoFrame;
+    let frameWidth: number, frameHeight: number;
+    switch (path) {
+      case 'HTMLVideoElement': {
+        const video = await getVideoElementFromCamera(t, requestSize?.[0], requestSize?.[1]);
+        // Pause the video so we get consistent readbacks. Wait a bit to be sure.
+        video.pause();
+        await resolveOnTimeout(10);
path: src/webgpu/web_platform/external_texture/video.spec.ts
user: login: kainino0x
body: On further thought maybe I did not succeed in waiting for the next frame. The camera should only be 30 or _maybe_ 60fps, so 10ms can't be more than one frame. I will try this again when I have time. Skipping one or two frames should be more reliable.
created_at: 2025-04-22T02:12:55Z
author_association: COLLABORATOR
diff_hunk: @@ -633,34 +636,63 @@ compared with 2d canvas rendering result.
 `
   )
   .params(u =>
+    // NOTE: Be careful not to add too many parameters here, this test is SLOW to initialize!
     u //
       .combineWithParams(checkNonStandardIsZeroCopyIfAvailable())
+      .combine('path', ['HTMLVideoElement', 'MediaStreamTrackProcessor'] as const)
       .combine('dstColorSpace', kPredefinedColorSpace)
+      .combine('requestSize', [undefined, [64, 48], [100, 300]] as const)
   )
   .fn(async t => {
-    const { dstColorSpace } = t.params;
-
-    const frame = await captureCameraFrame(t);
-
-    if (frame.displayHeight === 0 || frame.displayWidth === 0) {
-      t.skip('Captured video frame has 0 height or width.');
+    const { path, dstColorSpace, requestSize } = t.params;
+
+    // Enable this while debugging to show the "actual" and "expected" canvases on screen.
+    const kDebugShowCanvasesOnScreen = false;
+
+    let source: HTMLVideoElement | VideoFrame;
+    let frameWidth: number, frameHeight: number;
+    switch (path) {
+      case 'HTMLVideoElement': {
+        const video = await getVideoElementFromCamera(t, requestSize?.[0], requestSize?.[1]);
+        // Pause the video so we get consistent readbacks. Wait a bit to be sure.
+        video.pause();
+        await resolveOnTimeout(10);
path: src/webgpu/web_platform/external_texture/video.spec.ts
user: login: kainino0x
body: Oh wait, I got confused, I was thinking of the 200ms wait in getVideoElementFromCamera. That one is to get a real frame.

This wait is just a precaution to make sure the pause has time to take effect (since videos are controlled asynchronously). It was not proven to be needed for anything, I have no idea if it matters.
created_at: 2025-04-23T02:43:59Z
author_association: COLLABORATOR
diff_hunk: @@ -633,34 +636,63 @@ compared with 2d canvas rendering result.
 `
   )
   .params(u =>
+    // NOTE: Be careful not to add too many parameters here, this test is SLOW to initialize!
     u //
       .combineWithParams(checkNonStandardIsZeroCopyIfAvailable())
+      .combine('path', ['HTMLVideoElement', 'MediaStreamTrackProcessor'] as const)
       .combine('dstColorSpace', kPredefinedColorSpace)
+      .combine('requestSize', [undefined, [64, 48], [100, 300]] as const)
   )
   .fn(async t => {
-    const { dstColorSpace } = t.params;
-
-    const frame = await captureCameraFrame(t);
-
-    if (frame.displayHeight === 0 || frame.displayWidth === 0) {
-      t.skip('Captured video frame has 0 height or width.');
+    const { path, dstColorSpace, requestSize } = t.params;
+
+    // Enable this while debugging to show the "actual" and "expected" canvases on screen.
+    const kDebugShowCanvasesOnScreen = false;
+
+    let source: HTMLVideoElement | VideoFrame;
+    let frameWidth: number, frameHeight: number;
+    switch (path) {
+      case 'HTMLVideoElement': {
+        const video = await getVideoElementFromCamera(t, requestSize?.[0], requestSize?.[1]);
+        // Pause the video so we get consistent readbacks. Wait a bit to be sure.
+        video.pause();
+        await resolveOnTimeout(10);
path: src/webgpu/web_platform/external_texture/video.spec.ts
user: login: kainino0x
body: Did this properly with an onpause handler.
created_at: 2025-04-23T03:14:55Z
author_association: COLLABORATOR
diff_hunk: @@ -692,12 +727,22 @@ compared with 2d canvas rendering result.
     }
 
     const ctx = canvasContext as CanvasRenderingContext2D;
-    ctx.drawImage(frame, 0, 0, frameWidth, frameHeight);
+    ctx.drawImage(source, 0, 0, frameWidth, frameHeight);
 
     const imageData = ctx.getImageData(0, 0, frameWidth, frameHeight, {
       colorSpace: dstColorSpace,
     });
 
+    // Make sure the expected frame is not all black.
+    let isAllBlack = true;
+    for (let i = 0; i < imageData.data.length; i += 4) {
+      if (imageData.data[i] > 2 || imageData.data[i + 1] > 2 || imageData.data[i + 2] > 2) {
+        isAllBlack = false;
+        break;
+      }
+    }
+    t.expect(!isAllBlack, 'Video frame should not be all black');
path: src/webgpu/web_platform/external_texture/video.spec.ts
user: login: kainino0x
body: Done, thank you for the suggestion!
created_at: 2025-04-23T03:16:06Z
author_association: COLLABORATOR
diff_hunk: @@ -633,34 +636,63 @@ compared with 2d canvas rendering result.
 `
   )
   .params(u =>
+    // NOTE: Be careful not to add too many parameters here, this test is SLOW to initialize!
     u //
       .combineWithParams(checkNonStandardIsZeroCopyIfAvailable())
+      .combine('path', ['HTMLVideoElement', 'MediaStreamTrackProcessor'] as const)
       .combine('dstColorSpace', kPredefinedColorSpace)
+      .combine('requestSize', [undefined, [64, 48], [100, 300]] as const)
   )
   .fn(async t => {
-    const { dstColorSpace } = t.params;
-
-    const frame = await captureCameraFrame(t);
-
-    if (frame.displayHeight === 0 || frame.displayWidth === 0) {
-      t.skip('Captured video frame has 0 height or width.');
+    const { path, dstColorSpace, requestSize } = t.params;
+
+    // Enable this while debugging to show the "actual" and "expected" canvases on screen.
+    const kDebugShowCanvasesOnScreen = false;
+
+    let source: HTMLVideoElement | VideoFrame;
+    let frameWidth: number, frameHeight: number;
+    switch (path) {
+      case 'HTMLVideoElement': {
+        const video = await getVideoElementFromCamera(t, requestSize?.[0], requestSize?.[1]);
+        // Pause the video so we get consistent readbacks. Wait a bit to be sure.
+        video.pause();
+        await resolveOnTimeout(10);
path: src/webgpu/web_platform/external_texture/video.spec.ts
user: login: kainino0x
body: Actually... that didn't work. I guess I didn't realize this wait was necessary, but it is, the test fails without it.
Not sure why waiting on onpause doesn't do it though...
created_at: 2025-04-23T03:16:44Z
author_association: COLLABORATOR
diff_hunk: @@ -558,60 +563,85 @@ function callbackHelper(
   return { promise, callbackAndResolve: callbackAndResolve! };
 }
 
-/**
- * Create VideoFrame from camera captured frame. Check whether browser environment has
- * camera supported.
- * Returns a webcodec VideoFrame.
- *
- * @param test: GPUTest that requires getting VideoFrame
- *
- */
-export async function captureCameraFrame(test: GPUTest): Promise<VideoFrame> {
+async function getStreamFromCamera(
+  test: Fixture,
+  width: number | undefined,
+  height: number | undefined
+): Promise<MediaStream> {
   test.skipIf(typeof navigator === 'undefined', 'navigator does not exist in this environment');
   test.skipIf(
     typeof navigator.mediaDevices === 'undefined' ||
       typeof navigator.mediaDevices.getUserMedia === 'undefined',
     "Browser doesn't support capture frame from camera."
   );
 
-  const stream = await navigator.mediaDevices.getUserMedia({ video: true });
-  const track = stream.getVideoTracks()[0] as MediaStreamVideoTrack;
+  const stream = await navigator.mediaDevices.getUserMedia({
+    audio: false,
+    video: { width, height },
+  });
+  test.trackForCleanup({
+    close() {
+      for (const track of stream.getTracks()) {
+        track.stop();
+      }
+    },
+  });
+  return stream;
+}
+
+/**
+ * Uses MediaStreamTrackProcessor to capture a VideoFrame from the camera.
+ * Skips the test if not supported.
+ */
path: src/webgpu/web_platform/util.ts
user: login: kainino0x
body: Ah, usually I don't bother to document params, but in this case it was unclear what they were for. Fixed and also refactored a bit.
created_at: 2025-04-23T03:37:51Z
author_association: COLLABORATOR
diff_hunk: @@ -633,34 +636,63 @@ compared with 2d canvas rendering result.
 `
   )
   .params(u =>
+    // NOTE: Be careful not to add too many parameters here, this test is SLOW to initialize!
     u //
       .combineWithParams(checkNonStandardIsZeroCopyIfAvailable())
+      .combine('path', ['HTMLVideoElement', 'MediaStreamTrackProcessor'] as const)
       .combine('dstColorSpace', kPredefinedColorSpace)
+      .combine('requestSize', [undefined, [64, 48], [100, 300]] as const)
   )
   .fn(async t => {
-    const { dstColorSpace } = t.params;
-
-    const frame = await captureCameraFrame(t);
-
-    if (frame.displayHeight === 0 || frame.displayWidth === 0) {
-      t.skip('Captured video frame has 0 height or width.');
+    const { path, dstColorSpace, requestSize } = t.params;
+
+    // Enable this while debugging to show the "actual" and "expected" canvases on screen.
+    const kDebugShowCanvasesOnScreen = false;
+
+    let source: HTMLVideoElement | VideoFrame;
+    let frameWidth: number, frameHeight: number;
+    switch (path) {
+      case 'HTMLVideoElement': {
+        const video = await getVideoElementFromCamera(t, requestSize?.[0], requestSize?.[1]);
+        // Pause the video so we get consistent readbacks. Wait a bit to be sure.
+        video.pause();
+        await resolveOnTimeout(10);
path: src/webgpu/web_platform/external_texture/video.spec.ts
user: login: kainino0x
body: I misdiagnosed the issue, onpause is working fine. The diff I was seeing was just due to the color shift (https://crbug.com/411656657).
created_at: 2025-04-23T05:10:16Z
author_association: COLLABORATOR

PR comments:
user: login: kainino0x
created_at: 2025-04-23T05:11:46Z
body: Oops, wait, I still haven't fixed the other code path.
author_association: COLLABORATOR
user: login: kainino0x
created_at: 2025-04-23T05:34:17Z
body: Done, @shaoboyan091 PTAL. Thanks for keeping me honest in using proper logic instead of timeout hacks.
author_association: COLLABORATOR