Repository: wandb/weave
Pull request #4436: docs(weave): Document mp4 support
Author: login: J2-D2-3PO
Created: 2025-05-07T22:37:53Z | Merged: 2025-05-09T01:52:59Z

## Description

<!--
Include reference to internal ticket "Fixes WB-NNNNN" and/or GitHub issue "Fixes #NNNN" (if applicable)
-->

- Fixes https://wandb.atlassian.net/browse/DOCS-1486

Documents mp4 support

## Testing

1. yarn start on local
2. Tested and updated original code sample in a python venv with `weave` and `moviepy==1.0.3`



Diff:
diff --git a/docs/docs/guides/tracking/imgs/video-trace-popout.png b/docs/docs/guides/tracking/imgs/video-trace-popout.png
new file mode 100644
index 000000000000..0a94ac3ce2cc
Binary files /dev/null and b/docs/docs/guides/tracking/imgs/video-trace-popout.png differ
diff --git a/docs/docs/guides/tracking/imgs/video-trace.png b/docs/docs/guides/tracking/imgs/video-trace.png
new file mode 100644
index 000000000000..815fdf96f9ab
Binary files /dev/null and b/docs/docs/guides/tracking/imgs/video-trace.png differ
diff --git a/docs/docs/guides/tracking/video.md b/docs/docs/guides/tracking/video.md
new file mode 100644
index 000000000000..03580e54b56d
--- /dev/null
+++ b/docs/docs/guides/tracking/video.md
@@ -0,0 +1,91 @@
+# Video Support in Weave
+
+Weave now supports video tracing using [`moviepy`](https://zulko.github.io/moviepy/). This allows you to pass video inputs and outputs to traced functions, and Weave will automatically handle uploading and storing video data.
+
+:::note
+Video support in Weave is currently only available in Python.
+:::
+
+## Supported video types
+
+Weave recognizes `moviepy` video clip objects, such as:
+
+- A `VideoFileClip` loaded from a video file
+- In-memory clips like `ImageClip`, `ColorClip`, and `TextClip`
+
+### Direct upload of file-based clips
+
+If your clip is a `VideoFileClip` and has a valid filename with a supported extension, Weave will upload the file directly.
+
+**Supported file extensions:**
+
+- `.mp4`
+- `.webm`
+- `.gif`
+
+### In-memory clip support
+
+If the video object is in memory (no file on disk), Weave will encode it as an `.mp4` file and handle the upload automatically. This applies to clips of the following type:
+
+- `ImageClip`
+- `ColorClip`
+- `TextClip`
+
+## Example: Trace a video function
+
+The following code sample demonstrates how to trace a video processing function in Weave. The code sample:
+
+1. Initializes a Weave project `video-test`.
+2. Defines a `get_video` function tracked as a `weave.op` that extracts a 1 second subclip of the loaded `VideoFileClip` as a `VideoClip`.
+3. Uploads and tracks the clip in Weave.
+4. Automatically generates a dummy MP4 video if none is found.
+
+Before you can use the code sample, complete the prerequisites:
+
+1. Install `weave` and `moviepy==1.0.3`.
+2. Create a W&B account.
+
+:::important
+To avoid thread-safety issues, always pass the path to `VideoFileClip` objects instead of creating them outside the Weave `op`.
+:::
+
+```python
+import os
+import weave
+from moviepy import VideoFileClip, ColorClip, VideoClip
+
+# Update to your project name, or create a new project named 'video-test'
+weave.init('video-test')
+
+@weave.op
+def get_video(clip: VideoFileClip) -> VideoClip:
+    """Process a video by path rather than by passing the clip directly.
+
+    This ensures that the VideoFileClip is created and managed within the
+    Weave op's thread context, avoiding thread-safety issues.
+    """
+    new_clip = clip.subclip(0, 1)
+    return new_clip
+
+if __name__ == "__main__":
+    os.makedirs("videos", exist_ok=True)
+
+    # Update the path to point to your MP4 file
+    video_path = './videos/example.mp4'
+
+    # Generate a dummy video if it doesn't exist
+    # Dummy video contents: A red square that displays for 5 seconds
+    if not os.path.isfile(video_path):
+        print("No video found. Creating dummy video...")
+        dummy_clip = ColorClip(size=(640, 480), color=(255, 0, 0), duration=5)
+        dummy_clip.write_videofile(video_path, fps=24)
+
+    clip = VideoFileClip(video_path, has_mask=False, audio=True)
+    get_video(clip) 
+```
+
+When the code sample runs successfully, you can view your video by clicking the link in the **Traces** table of your project.
+
+![A trace of a video processing function in the Traces table.](imgs/video-trace.png)
+
+![An MP4 video uploaded to Weave, viewed in the Traces popout.](imgs/video-trace-popout.png)
diff --git a/docs/sidebars.ts b/docs/sidebars.ts
index 12d215f92013..c2b4e6b53701 100644
--- a/docs/sidebars.ts
+++ b/docs/sidebars.ts
@@ -59,6 +59,7 @@ const sidebars: SidebarsConfig = {
             },
             "guides/tracking/trace-tree",
             "guides/tracking/otel",
+            "guides/tracking/video"
           ]
         },
         {


Reviews:
user: login: J2-D2-3PO
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T22:41:39Z
user: login: zbirenbaum
body: One small note on the code and also it would be good to show the video player in the trace view in the screenshot rather than just the trace table
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T22:54:28Z
user: login: J2-D2-3PO
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T23:21:05Z
user: login: J2-D2-3PO
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T23:23:15Z
user: login: jamie-rasmussen
body: Thoughts on having this as a separate page vs. with audio and images on https://weave-docs.wandb.ai/guides/core-types/media ?
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-08T00:02:20Z
user: login: zbirenbaum
body: Looks awesome, thank you!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T01:09:39Z

Review comments:
diff_hunk: @@ -0,0 +1,89 @@
+# Video Support in Weave
+
+Weave now supports video tracing using [`moviepy`](https://zulko.github.io/moviepy/). This allows you to pass video inputs and outputs to traced functions, and Weave will automatically handle uploading and storing video data.
+
+:::note
+Video support in Weave is currently only available in Python.
+:::
+
+## Supported video types
+
+Weave recognizes `moviepy` video clip objects, such as:
+
+- A `VideoFileClip` loaded from a video file
+- In-memory clips like `ImageClip`, `ColorClip`, and `TextClip`
+
+### Direct upload of file-based clips
+
+If your clip is a `VideoFileClip` and has a valid filename with a supported extension, Weave will upload the file directly.
+
+**Supported file extensions:**
+
+- `.mp4`
+- `.webm`
+- `.gif`
+
+### In-memory clip support
+
+If the video object is in memory (no file on disk), Weave will encode it as an `.mp4` file and handle the upload automatically. This applies to clips of the following type:
+
+- `ImageClip`
+- `ColorClip`
+- `TextClip`
+
+## Example: Trace a video function
+
+The following code sample demonstrates how to trace a video processing function in Weave. The code sample:
+
+1. Initializes a Weave project `video-test`.
+2. Defines a `get_video` function tracked as a `weave.op` that extracts a 5-second subclip of the loaded `VideoFileClip` as a `VideoClip`.
+3. Uploads and tracks the clip in Weave.
+4. Automatically generates a dummy MP4 video if none is found.
+
+Before you can use the code sample, complete the prerequisites:
+
+1. Install `weave` and `moviepy==1.0.3`.
+2. Create a W&B account 
path: docs/docs/guides/tracking/video.md
user: login: J2-D2-3PO
body: ```suggestion
2. Create a W&B account.
```
created_at: 2025-05-07T22:41:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,89 @@
+# Video Support in Weave
+
+Weave now supports video tracing using [`moviepy`](https://zulko.github.io/moviepy/). This allows you to pass video inputs and outputs to traced functions, and Weave will automatically handle uploading and storing video data.
+
+:::note
+Video support in Weave is currently only available in Python.
+:::
+
+## Supported video types
+
+Weave recognizes `moviepy` video clip objects, such as:
+
+- A `VideoFileClip` loaded from a video file
+- In-memory clips like `ImageClip`, `ColorClip`, and `TextClip`
+
+### Direct upload of file-based clips
+
+If your clip is a `VideoFileClip` and has a valid filename with a supported extension, Weave will upload the file directly.
+
+**Supported file extensions:**
+
+- `.mp4`
+- `.webm`
+- `.gif`
+
+### In-memory clip support
+
+If the video object is in memory (no file on disk), Weave will encode it as an `.mp4` file and handle the upload automatically. This applies to clips of the following type:
+
+- `ImageClip`
+- `ColorClip`
+- `TextClip`
+
+## Example: Trace a video function
+
+The following code sample demonstrates how to trace a video processing function in Weave. The code sample:
+
+1. Initializes a Weave project `video-test`.
+2. Defines a `get_video` function tracked as a `weave.op` that extracts a 5-second subclip of the loaded `VideoFileClip` as a `VideoClip`.
+3. Uploads and tracks the clip in Weave.
+4. Automatically generates a dummy MP4 video if none is found.
+
+Before you can use the code sample, complete the prerequisites:
+
+1. Install `weave` and `moviepy==1.0.3`.
+2. Create a W&B account.
+
+:::important
+To avoid thread-safety issues, always pass the path to `VideoFileClip` objects instead of creating them outside the Weave `op`.
+:::
+
+```python
+import os
+import weave
+from moviepy import VideoFileClip, ColorClip, VideoClip
+
+# Update to your project name, or create a new project named 'video-test'
+weave.init('video-test')
+
+@weave.op
+def get_video(clip: VideoFileClip) -> VideoClip:
+    """Process a video by path rather than by passing the clip directly.
+
+    This ensures that the VideoFileClip is created and managed within the
+    Weave op's thread context, avoiding thread-safety issues.
+    """
+    new_clip = clip.subclipped(0, 5)
path: docs/docs/guides/tracking/video.md
user: login: zbirenbaum
body: Lets make this `clip.subclipped(0, 1)` so that the dummy clip and original clip differ
created_at: 2025-05-07T22:52:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,91 @@
+# Video Support in Weave
+
+Weave now supports video tracing using [`moviepy`](https://zulko.github.io/moviepy/). This allows you to pass video inputs and outputs to traced functions, and Weave will automatically handle uploading and storing video data.
+
+:::note
+Video support in Weave is currently only available in Python.
+:::
+
+## Supported video types
+
+Weave recognizes `moviepy` video clip objects, such as:
+
+- A `VideoFileClip` loaded from a video file
+- In-memory clips like `ImageClip`, `ColorClip`, and `TextClip`
+
+### Direct upload of file-based clips
+
+If your clip is a `VideoFileClip` and has a valid filename with a supported extension, Weave will upload the file directly.
+
+**Supported file extensions:**
+
+- `.mp4`
+- `.webm`
+- `.gif`
+
+### In-memory clip support
+
+If the video object is in memory (no file on disk), Weave will encode it as an `.mp4` file and handle the upload automatically. This applies to clips of the following type:
+
+- `ImageClip`
+- `ColorClip`
+- `TextClip`
+
+## Example: Trace a video function
+
+The following code sample demonstrates how to trace a video processing function in Weave. The code sample:
+
+1. Initializes a Weave project `video-test`.
+2. Defines a `get_video` function tracked as a `weave.op` that extracts a 1 second subclip of the loaded `VideoFileClip` as a `VideoClip`.
+3. Uploads and tracks the clip in Weave.
+4. Automatically generates a dummy MP4 video if none is found.
+
+Before you can use the code sample, complete the prerequisites:
+
+1. Install `weave` and `moviepy==1.0.3`.
+2. Create a W&B account.
+
+:::important
+To avoid thread-safety issues, always pass the path to `VideoFileClip` objects instead of creating them outside the Weave `op`.
+:::
+
+```python
+import os
+import weave
+from moviepy import VideoFileClip, ColorClip, VideoClip
+
+# Update to your project name, or create a new project named 'video-test'
+weave.init('video-test')
+
+@weave.op
+def get_video(clip: VideoFileClip) -> VideoClip:
+    """Process a video by path rather than by passing the clip directly.
+
+    This ensures that the VideoFileClip is created and managed within the
+    Weave op's thread context, avoiding thread-safety issues.
+    """
+    new_clip = clip.subclipped(0, 1)
path: docs/docs/guides/tracking/video.md
user: login: J2-D2-3PO
body: ```suggestion
    new_clip = clip.subclip(0, 1)
```
created_at: 2025-05-07T23:21:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,89 @@
+# Video Support in Weave
+
+Weave now supports video tracing using [`moviepy`](https://zulko.github.io/moviepy/). This allows you to pass video inputs and outputs to traced functions, and Weave will automatically handle uploading and storing video data.
+
+:::note
+Video support in Weave is currently only available in Python.
+:::
+
+## Supported video types
+
+Weave recognizes `moviepy` video clip objects, such as:
+
+- A `VideoFileClip` loaded from a video file
+- In-memory clips like `ImageClip`, `ColorClip`, and `TextClip`
+
+### Direct upload of file-based clips
+
+If your clip is a `VideoFileClip` and has a valid filename with a supported extension, Weave will upload the file directly.
+
+**Supported file extensions:**
+
+- `.mp4`
+- `.webm`
+- `.gif`
+
+### In-memory clip support
+
+If the video object is in memory (no file on disk), Weave will encode it as an `.mp4` file and handle the upload automatically. This applies to clips of the following type:
+
+- `ImageClip`
+- `ColorClip`
+- `TextClip`
+
+## Example: Trace a video function
+
+The following code sample demonstrates how to trace a video processing function in Weave. The code sample:
+
+1. Initializes a Weave project `video-test`.
+2. Defines a `get_video` function tracked as a `weave.op` that extracts a 5-second subclip of the loaded `VideoFileClip` as a `VideoClip`.
+3. Uploads and tracks the clip in Weave.
+4. Automatically generates a dummy MP4 video if none is found.
+
+Before you can use the code sample, complete the prerequisites:
+
+1. Install `weave` and `moviepy==1.0.3`.
+2. Create a W&B account.
+
+:::important
+To avoid thread-safety issues, always pass the path to `VideoFileClip` objects instead of creating them outside the Weave `op`.
+:::
+
+```python
+import os
+import weave
+from moviepy import VideoFileClip, ColorClip, VideoClip
+
+# Update to your project name, or create a new project named 'video-test'
+weave.init('video-test')
+
+@weave.op
+def get_video(clip: VideoFileClip) -> VideoClip:
+    """Process a video by path rather than by passing the clip directly.
+
+    This ensures that the VideoFileClip is created and managed within the
+    Weave op's thread context, avoiding thread-safety issues.
+    """
+    new_clip = clip.subclipped(0, 5)
path: docs/docs/guides/tracking/video.md
user: login: J2-D2-3PO
body: Updated, also realized that the code was using the function from 2.1.2 (subclipped) from my testing, updated it to use the 1.0.3. function (subclip) https://github.com/wandb/weave/pull/4436/files#diff-ea9928fd306380baddb60acc9456f702ff0a27b4b439ed3033819dc504fdc50fR67
created_at: 2025-05-07T23:23:15Z
author_association: CONTRIBUTOR

PR comments:
user: login: zbirenbaum
created_at: 2025-05-07T23:39:13Z
body: It looks like something is going on with the Video Player in the trace view, I'm looking into it
author_association: CONTRIBUTOR
user: login: J2-D2-3PO
created_at: 2025-05-08T17:09:14Z
body: > Thoughts on having this as a separate page vs. with audio and images on https://weave-docs.wandb.ai/guides/core-types/media ?

I get the idea, but I feel like that page may get bloated quickly, since the video support page is it's own full page worth of contents and it sounds like we are going to have more audio support soon. I also want to make sure users can find what they are looking for (e.g. tracing video info from the Tracing subsection) 

If anything, in the future we might want to break "Logging Media" into a grouping of multiple pages (Audio, Video, Images) each with their own intro page, examples, etc.. Which sounds like it's getting into IA refactor territory again... 😅 so I vote let's just ship this as is for now
author_association: CONTRIBUTOR
user: login: J2-D2-3PO
created_at: 2025-05-08T22:00:15Z
body: @zbirenbaum I think I've addressed everything here, lmk if / when this is good to merge
author_association: CONTRIBUTOR