Repository: gradio-app/gradio
Pull request: enhancement: record audio and video from webcam simultaneously
# Description

From @MandarGogate issue:
> **Is your feature request related to a problem? Please describe.**
I want to build a demo for machine learning model that processes Audio-Visual data from webcam. The current Video component records from webcam without audio.
> **Describe the solution you'd like**
Since the Video component uses MediaRecorder API it can record audio and video simultaneously.

Changes:
```
- Enables audio in Webcam.svelte inside `access_webcam` function
```

Closes: #2704

# Checklist:

- [x] I have performed a review of the code
- [ ] I have added a short summary of my change to the CHANGELOG.md
- [x] The code follows the style guidelines of this project

Problem statement:
Recording Audio and Video from webcam simultaneously
- [X] I have searched to see if a similar issue already exists.


**Is your feature request related to a problem? Please describe.**  
I want to build a demo for machine learning model that processes Audio-Visual data from webcam. The current Video component records from webcam without audio. 

**Describe the solution you'd like**  
Since the Video component uses MediaRecorder API it can record audio and video simultaneously.

Hints from issue discussion:
I managed to update the Webcam.svelte to fix this. You can have a look at [https://github.com/MandarGogate/gradio/commit/2d291e7f311192899e01b5f82133905ce1dace67](https://github.com/MandarGogate/gradio/commit/2d291e7f311192899e01b5f82133905ce1dace67). I tested this on Safari, Chrome and Firefox.
Great, thanks so much @MandarGogate! Would you be able to open a PR and happy to review it?

Commit message: Add include_audio option in Video component

Patch to review:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7fc0ce2791..b05ffdfbcd 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -19,6 +19,7 @@ No changes to highlight.
 * Make try examples button more prominent by [@aliabd](https://github.com/aliabd) in [PR 2705](https://github.com/gradio-app/gradio/pull/2705)
 * Fix id clashes in docs by [@aliabd](https://github.com/aliabd) in [PR 2713](https://github.com/gradio-app/gradio/pull/2713)
 * Fix typos in guide docs by [@andridns](https://github.com/andridns) in [PR 2722](https://github.com/gradio-app/gradio/pull/2722)
+* Add include_audio option in Video component by [@mandargogate](https://github.com/MandarGogate) in [PR 2721](https://github.com/gradio-app/gradio/pull/2721)
 
 ## Contributors Shoutout:
 * [@andridns](https://github.com/andridns) made their first contribution in [PR 2722](https://github.com/gradio-app/gradio/pull/2722)!
diff --git a/gradio/components.py b/gradio/components.py
index 08b52b3023..2fcbe9dff4 100644
--- a/gradio/components.py
+++ b/gradio/components.py
@@ -1600,6 +1600,7 @@ def __init__(
         visible: bool = True,
         elem_id: Optional[str] = None,
         mirror_webcam: bool = True,
+        include_audio: bool = True,
         **kwargs,
     ):
         """
@@ -1612,7 +1613,8 @@ def __init__(
             interactive: if True, will allow users to upload a video; if False, can only be used to display videos. If not provided, this is inferred based on whether the component is used as an input or output.
             visible: If False, component will be hidden.
             elem_id: An optional string that is assigned as the id of this component in the HTML DOM. Can be used for targeting CSS styles.
-            mirror_webcam: If True webcma will be mirrored. Default is True.
+            mirror_webcam: If True webcam will be mirrored. Default is True.
+            include_audio: If False, component will only record/keep muted video.
         """
         self.temp_dir = tempfile.mkdtemp()
         self.format = format
@@ -1623,6 +1625,7 @@ def __init__(
             )
         self.source = source
         self.mirror_webcam = mirror_webcam
+        self.include_audio = include_audio
         IOComponent.__init__(
             self,
             label=label,
@@ -1639,6 +1642,7 @@ def get_config(self):
             "source": self.source,
             "value": self.value,
             "mirror_webcam": self.mirror_webcam,
+            "include_audio": self.include_audio,
             **IOComponent.get_config(self),
         }
 
@@ -1691,7 +1695,8 @@ def preprocess(self, x: Dict[str, str] | None) -> str | None:
         flip = self.source == "webcam" and self.mirror_webcam
         if modify_format or flip:
             format = f".{self.format if modify_format else uploaded_format}"
-            output_options = ["-vf", "hflip", "-c:a", "copy"] if flip else None
+            output_options = ["-vf", "hflip", "-c:a", "copy"] if flip else []
+            output_options += ["-an"] if not self.include_audio else []
             flip_suffix = "_flip" if flip else ""
             output_file_name = str(
                 file_name.with_name(f"{file_name.stem}{flip_suffix}{format}")
@@ -1702,6 +1707,14 @@ def preprocess(self, x: Dict[str, str] | None) -> str | None:
             )
             ff.run()
             return output_file_name
+        elif not self.include_audio:
+            output_file_name = str(file_name.with_name(f"muted_{file_name.name}"))
+            ff = FFmpeg(
+                inputs={str(file_name): None},
+                outputs={output_file_name: ["-an"]},
+            )
+            ff.run()
+            return output_file_name
         else:
             return str(file_name)
 
diff --git a/gradio/templates.py b/gradio/templates.py
index 3e5ee8fe72..3a95cec3f8 100644
--- a/gradio/templates.py
+++ b/gradio/templates.py
@@ -333,6 +333,7 @@ def __init__(
         visible: bool = True,
         elem_id: Optional[str] = None,
         mirror_webcam: bool = True,
+        include_audio: bool = True,
         **kwargs,
     ):
         super().__init__(
@@ -345,6 +346,7 @@ def __init__(
             visible=visible,
             elem_id=elem_id,
             mirror_webcam=mirror_webcam,
+            include_audio=include_audio,
             **kwargs,
         )
 
diff --git a/ui/packages/app/src/components/Video/Video.svelte b/ui/packages/app/src/components/Video/Video.svelte
index d88f0bfef2..761b5e1be8 100644
--- a/ui/packages/app/src/components/Video/Video.svelte
+++ b/ui/packages/app/src/components/Video/Video.svelte
@@ -20,6 +20,7 @@
 	export let loading_status: LoadingStatus;
 	export let style: Styles = {};
 	export let mirror_webcam: boolean;
+	export let include_audio: boolean;
 
 	export let mode: "static" | "dynamic";
 
@@ -60,6 +61,7 @@
 			or_text={$_("or")}
 			upload_text={$_("interface.click_to_upload")}
 			{mirror_webcam}
+			{include_audio}
 			on:change
 			on:clear
 			on:play
diff --git a/ui/packages/image/src/Webcam.svelte b/ui/packages/image/src/Webcam.svelte
index 0911405732..841a347a64 100644
--- a/ui/packages/image/src/Webcam.svelte
+++ b/ui/packages/image/src/Webcam.svelte
@@ -9,6 +9,7 @@
 
 	export let mode: "image" | "video" = "image";
 	export let mirror_webcam: boolean;
+	export let include_audio: boolean;
 
 	const dispatch = createEventDispatcher();
 
@@ -17,9 +18,11 @@
 	async function access_webcam() {
 		try {
 			stream = await navigator.mediaDevices.getUserMedia({
-				video: true
+				video: true,
+				audio: include_audio
 			});
 			video_source.srcObject = stream;
+			video_source.muted = true;
 			video_source.play();
 		} catch (err) {
 			if (err instanceof DOMException && err.name == "NotAllowedError") {
diff --git a/ui/packages/video/src/Video.svelte b/ui/packages/video/src/Video.svelte
index 6351e7de7a..c300f62aa8 100644
--- a/ui/packages/video/src/Video.svelte
+++ b/ui/packages/video/src/Video.svelte
@@ -14,6 +14,7 @@
 	export let label: string | undefined = undefined;
 	export let show_label: boolean;
 	export let mirror_webcam: boolean;
+	export let include_audio: boolean;
 
 	export let drop_text: string = "Drop a video file";
 	export let or_text: string = "or";
@@ -63,6 +64,7 @@
 	{:else if source === "webcam"}
 		<Webcam
 			{mirror_webcam}
+			{include_audio}
 			mode="video"
 			on:error
 			on:capture={({ detail }) => dispatch("change", detail)}


Code review comments:
diff_hunk: @@ -1612,7 +1613,8 @@ def __init__(
             interactive: if True, will allow users to upload a video; if False, can only be used to display videos. If not provided, this is inferred based on whether the component is used as an input or output.
             visible: If False, component will be hidden.
             elem_id: An optional string that is assigned as the id of this component in the HTML DOM. Can be used for targeting CSS styles.
-            mirror_webcam: If True webcma will be mirrored. Default is True.
+            mirror_webcam: If True webcam will be mirrored. Default is True.
+            include_audio: If False, component will only record/keep muted video.
path: gradio/components.py
text: @user1:
```suggestion
            include_audio: If False, component will not record or retain the audio track for a video.
```