Repository: gradio-app/gradio

Issue:
Add option for `autoplay` in Audio component
- [x] I have searched to see if a similar issue already exists.


**Is your feature request related to a problem? Please describe.**  
I'd like to be able to auto play a sound when it comes back from the inference.

**Describe the solution you'd like**  
An new attribute, `False` by default, in the `Audio` component to enable/disable autoplay of the HTML audio element.



Discussion:
Seems reasonable to me. I think this would make sense as a parameter in the "style()" function of the "Audio()" component.
Auto playing audio is not allowed in chrome, although works in other browsers. Auto playing videos work in all browsers as long as the video is muted. 

There are some workarounds but they are pretty heavy and can be inconsistent. 

Will see if we can any other options here since there have already been user interactions. 
I would recommend against using any work around to enforce auto play and to use the HTML `autoplay` attribute instead and just let the browsers do what they will do. Audio and video autoplay can cause accessibility issues, especially for users who are using screen readers. If you use `autoplay` and ensure there is a way for users to pause audio and video, it shouldn’t cause any a11y issues.

http://www.w3.org/TR/WCAG20-TECHS/F93.html
In most cases, it is a slightly different case to normal autoplaying videos. It is similar to a user uploading a video via a file input and that video then autoplaying (as opposed to autoplaying on page load) but the main issue still stands. We would need to focus an appropriate part of the screen (possibly the video elements itself).

On balance, I think autoplaying with the video muted (chrome default) is probably the best approach if we decide to support this.

> I think this would make sense as a parameter in the "style()" function of the "Audio()" component.

I don't agree with this though, this isn't a stylistic choice but changes the behaviour of the UI. It should be a kwarg and updatable via `gr.Update` imo.
I needed this for a small internal demo, where we wanted to output text-to-speech without the user pressing an additional button. My workaround looks like this:

```python
    audio_el = gr.Audio(type="numpy", elem_id="speaker")
    autoplay_audio = """async () => {
        console.log('playing audio in 2 seconds')
        let gradioEl = document.querySelector('body > gradio-app').shadowRoot;
        setTimeout(() => {
            let audioplayer = gradioEl.querySelector('#speaker > audio');
            audioplayer.play();
        }, 2000)
        }"""
    transcription.change(fn=update_audio, inputs=state, outputs=audio_el, _js=autoplay_audio)
```

It works fine, but obviously the 2 seconds delay are hard coded because this is how long our system approximately needs to display the audio player. Is there a way to get the state of the audio component?
> I needed this for a small internal demo, where we wanted to output text-to-speech without the user pressing an additional button. My workaround looks like this:
> 
> ```python
>     audio_el = gr.Audio(type="numpy", elem_id="speaker")
>     autoplay_audio = """async () => {
>         console.log('playing audio in 2 seconds')
>         let gradioEl = document.querySelector('body > gradio-app').shadowRoot;
>         setTimeout(() => {
>             let audioplayer = gradioEl.querySelector('#speaker > audio');
>             audioplayer.play();
>         }, 2000)
>         }"""
>     transcription.change(fn=update_audio, inputs=state, outputs=audio_el, _js=autoplay_audio)
> ```
> 
> It works fine, but obviously the 2 seconds delay are hard coded because this is how long our system approximately needs to display the audio player. Is there a way to get the state of the audio component?

I'm in same satuation to showcase an ai chat demo with TTS feature and no user.
Agreeed. It would be great to enable this, as well as https://github.com/gradio-app/gradio/discussions/3316. cc @aliabid94 
@pi43r and/or @Arcadia822 it sounds like you got a workaround running? @pi43r thank you for the code snippet. I unfortunately am having trouble getting it to work. I either end up with a `None` input, or I just don't see any of the javascript running. Did you have to configure your browser, or use a specific browser, to get it working? Can you describe what `update_audio` and `state` is in that code so I can understand what is feeding into `audio_el`?
> @pi43r and/or @Arcadia822 it sounds like you got a workaround running? @pi43r thank you for the code snippet. I unfortunately am having trouble getting it to work. I either end up with a `None` input, or I just don't see any of the javascript running. Did you have to configure your browser, or use a specific browser, to get it working? Can you describe what `update_audio` and `state` is in that code so I can understand what is feeding into `audio_el`?

It's been a while since I wrote the code and gradio might have changed. It should work in any browser (maybe not in safari because apple is different).
`update_audio` in our case was a simple function that generates the audio and converts it to a numpy array, then places it in the `state` list.
https://gradio.app/interface-state/

I might be able to make a minimal reproduction in the future on huggingface...
Strictly as a quick workaround, when I launched the interface of a chatbot I was playing with, I set debug=True and  the audio played as the text was returned.  

An audio player box popped up below the gradio input/output boxes, but the audio played automatically.   

I also suppressed all warnings, to keep them from displaying.    
@robjm16 
> Strictly as a quick workaround, when I launched the interface of a chatbot I was playing with, I set debug=True and the audio played as the text was returned.

Thanks for the tip. I tried `debug=True` in both the launch function and in the audio block. No luck on my end. Though it may be because I'm using blocks instead? Where did you place your `debug=True`?
in launch function.

On Mon, Mar 13, 2023 at 8:38 PM Tom Szumowski ***@***.***>
wrote:

> @robjm16 <https://github.com/robjm16>
>
> Strictly as a quick workaround, when I launched the interface of a chatbot
> I was playing with, I set debug=True and the audio played as the text was
> returned.
>
> Thanks for the tip. I tried debug=True in both the launch function and in
> the audio block. No luck on my end. Though it may be because I'm using
> blocks instead? Where did you place your debug=True?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/gradio-app/gradio/issues/1349#issuecomment-1467171615>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AGFOR2GNCF2Y5SDAC7UWRU3W364ZNANCNFSM5WPTK5MA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>

I was trying to get this voice-enabled chatbot to work (code below).  Wasn't using gradio blocks. 
Most of the code taken from: 
https://github.com/hackingthemarkets/chatgpt-api-whisper-api-voice-assistant/blob/main/therapist.py


import openai
import gradio as gr
import gtts
from playsound import playsound
from gtts import gTTS #Import Google Text to Speech
from IPython.display import Audio #Import Audio method from IPython's Display Class

import warnings
warnings.filterwarnings("ignore")
openai.api_key = 'YOUR KEY HERE" 

# Create list of messages, starting with initial message to the system
messages = [{"role": "system", "content": 'You are a therapist. Respond to all input in 25 words or less.'}]

def transcribe(audio):
  """
  Transcribes the user's audio input using the OpenAI API,
  generates a response from the chatbot using GPT-3, converts the response into 
  speech using the gTTS library, updates the conversation history, and returns 
  the updated conversation history as a string.

  Parameters:
    audio (str): The filepath of the audio file containing the user's input.

  Returns:
  str: A string containing the updated conversation history, with each message formatted as "role: content" and separated by two newlines. 
  """
    # Declare messages a global variable (not local to the function)
    global messages  

    # Get user's audio, transcribe it and append it to messages 
    audio_file = open(audio, "rb") #"open" is built-in Python command
    transcript = openai.Audio.transcribe("whisper-1", audio_file)
    messages.append({"role": "user", "content": transcript["text"]})

    # Get the therapist's response, append to messages 
    response = openai.ChatCompletion.create(model="gpt-3.5-turbo", messages=messages)
    system_message = response["choices"][0]["message"]
    messages.append(system_message)

    # Create audio from therapist's text response   
    msg=system_message["content"]  
    # print(msg) # For validation 
    talk_file=make_into_speech(msg)
    display(Audio(talk_file, autoplay=True)) 

    # Update the rolling chat transcript 
    chat_transcript = ""
    for message in messages:
        if message['role'] != 'system':
            chat_transcript += message['role'] + ": " + message['content'] + "\n\n"

    return chat_transcript

def make_into_speech(words):
    """
    Takes a string as input, converts it to speech using the gTTS library, 
    saves the speech as a WAV file, and returns the filepath of the saved WAV file.    
    Parameters:
    - `words` (str): The input string to convert to speech.    
    Returns:
    - `sound_file` (str): The filepath of the saved WAV file.    
    Example:
    >>> make_into_speech('Hello, how are you today?')
    '2.wav'    
    The function converts the input string to speech and returns the filepath of the saved WAV file.
    """
  tts = gTTS(words) #Provide the string to convert to speech
  tts.save('2.wav') #Save the string converted to speech as a .wav file
  sound_file = '2.wav'
  return sound_file

# Launch the interface
ui = gr.Interface(fn=transcribe, inputs=gr.Audio(source="microphone", type="filepath", label="Record Here"), outputs=[gr.Text(label="Chat Transcript")])
ui.launch(debug=True) 
Thanks again for the additional info. I was able to get it to work for my use case with the usage of `_js` like described above, but a slightly different javascript string:
https://github.com/tszumowski/vocaltales_storyteller_chatbot/blob/46d46799ff8cdff2f016e484b0ade0e14cb12f8a/storyteller.py#L236-L242

```
    autoplay_audio = """
            async () => {{
                setTimeout(() => {{
                    document.querySelector('#speaker audio').play();
                }}, {speech_delay});
            }}
        """
```
I now use this method to add autoplay, which can be referred to below：
```
import gradio as gr
from gtts import gTTS
from io import BytesIO
import base64


def text_to_speech(text):
    tts = gTTS(text)
    tts.save('hello_world.mp3')

    audio_bytes = BytesIO()
    tts.write_to_fp(audio_bytes)
    audio_bytes.seek(0)

    audio = base64.b64encode(audio_bytes.read()).decode("utf-8")
    audio_player = f'<audio src="data:audio/mpeg;base64,{audio}" controls autoplay></audio>'

    return audio_player


with gr.Blocks() as demo:
    html = gr.HTML()
    # html.visible = False

    text = gr.Text()
    btn = gr.Button("OK")
    btn.click(text_to_speech, inputs=[text], outputs=[html])

demo.launch()
```
Thanks. This worked for me.

On Thu, Mar 30, 2023 at 5:52 AM CsqTom ***@***.***> wrote:

> I now use this method to add autoplay, which can be referred to below：
> `
> import gradio as gr
> from gtts import gTTS
> from io import BytesIO
> import base64
>
> def text_to_speech(text):
> tts = gTTS(text)
> tts.save('hello_world.mp3')
>
> audio_bytes = BytesIO()
> tts.write_to_fp(audio_bytes)
> audio_bytes.seek(0)
>
> audio = base64.b64encode(audio_bytes.read()).decode("utf-8")
> audio_player = f'<audio src="data:audio/mpeg;base64,{audio}" controls autoplay></audio>'
>
> return audio_player
>
> with gr.Blocks() as demo:
> html = gr.HTML()
> # html.visible = False
>
> text = gr.Text()
> btn = gr.Button("OK")
> btn.click(text_to_speech, inputs=[text], outputs=[html])
>
> demo.launch()
> `
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/gradio-app/gradio/issues/1349#issuecomment-1490017507>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AGFOR2EBUQLGNQ455SJW7B3W6VJUVANCNFSM5WPTK5MA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>

> I now use this method to add autoplay, which can be referred to below：
> 
> ```
> import gradio as gr
> from gtts import gTTS
> from io import BytesIO
> import base64
> 
> 
> def text_to_speech(text):
>     tts = gTTS(text)
>     tts.save('hello_world.mp3')
> 
>     audio_bytes = BytesIO()
>     tts.write_to_fp(audio_bytes)
>     audio_bytes.seek(0)
> 
>     audio = base64.b64encode(audio_bytes.read()).decode("utf-8")
>     audio_player = f'<audio src="data:audio/mpeg;base64,{audio}" controls autoplay></audio>'
> 
>     return audio_player
> 
> 
> with gr.Blocks() as demo:
>     html = gr.HTML()
>     # html.visible = False
> 
>     text = gr.Text()
>     btn = gr.Button("OK")
>     btn.click(text_to_speech, inputs=[text], outputs=[html])
> 
> demo.launch()
> ```
Thank you! It works!
and I am using .wav, so I got this:
```python
def audio_to_html(audio):
    audio_bytes = BytesIO()
    wavio.write(audio_bytes, audio[1].astype(np.float32), audio[0], sampwidth=4)
    audio_bytes.seek(0)

    audio_base64 = base64.b64encode(audio_bytes.read()).decode("utf-8")
    audio_player = f'<audio src="data:audio/mpeg;base64,{audio_base64}" controls autoplay></audio>'

    return audio_player
```
and adding:
```python
import wavio
```
Thanks the above solution worked for me, For a more beginner friendly description of what's going on:

- You have an audio, Consider it to be in bytes (or convert it to bytes)
- Convert it to bytesIO using `audio_io=BtesIO(audio_in_bytes)` (Library: `from io import BytesIO`)
- Point it to the beginning of the audio file using `audio_io.seek(0)`
- Convert the audio_io to base64 to be used in html command using `audio_base64 = base64.b64encode(audio_io.read()).decode("utf-8")` (Library `import base64`)
- Finally, write the html command as string that invokes autoplay for your audio as `audio_html = f'<audio src="data:audio/mpeg;base64,{audio_base64}" controls autoplay></audio>'`

Use the `audio_html` as your output in gradio to auto play that converted audio. This can be altered based on your usage
@CsqTom 
> I now use this method to add autoplay, which can be referred to below：

Thank you! Your code worked perfectly when I [integrated it into my codebase](https://github.com/tszumowski/vocaltales_storyteller_chatbot/pull/8), and cleaned it up a lot too! I appreciate the reproducible example you provided as well!  


Resolving patch:
diff --git a/gradio/components.py b/gradio/components.py
--- a/gradio/components.py
+++ b/gradio/components.py
@@ -2172,6 +2172,7 @@ def __init__(
         elem_classes: list[str] | str | None = None,
         mirror_webcam: bool = True,
         include_audio: bool | None = None,
+        autoplay: bool = False,
         **kwargs,
     ):
         """
@@ -2195,6 +2196,7 @@ def __init__(
             include_audio: Whether the component should record/retain the audio track for a video. By default, audio is excluded for webcam videos and included for uploaded videos.
         """
         self.format = format
+        self.autoplay = autoplay
         valid_sources = ["upload", "webcam"]
         if source not in valid_sources:
             raise ValueError(
@@ -2231,6 +2233,7 @@ def get_config(self):
             "width": self.width,
             "mirror_webcam": self.mirror_webcam,
             "include_audio": self.include_audio,
+            "autoplay": self.autoplay,
             **IOComponent.get_config(self),
         }
 
@@ -2250,6 +2253,7 @@ def update(
         min_width: int | None = None,
         interactive: bool | None = None,
         visible: bool | None = None,
+        autoplay: bool | None = None,
     ):
         return {
             "source": source,
@@ -2263,6 +2267,7 @@ def update(
             "interactive": interactive,
             "visible": visible,
             "value": value,
+            "autoplay": autoplay,
             "__type__": "update",
         }
 
@@ -2518,6 +2523,7 @@ def __init__(
         elem_id: str | None = None,
         elem_classes: list[str] | str | None = None,
         format: Literal["wav", "mp3"] = "wav",
+        autoplay: bool = False,
         **kwargs,
     ):
         """
@@ -2572,12 +2578,14 @@ def __init__(
         )
         TokenInterpretable.__init__(self)
         self.format = format
+        self.autoplay = autoplay
 
     def get_config(self):
         return {
             "source": self.source,
             "value": self.value,
             "streaming": self.streaming,
+            "autoplay": self.autoplay,
             **IOComponent.get_config(self),
         }
 
@@ -2598,6 +2606,7 @@ def update(
         min_width: int | None = None,
         interactive: bool | None = None,
         visible: bool | None = None,
+        autoplay: bool | None = None,
     ):
         return {
             "source": source,
@@ -2609,6 +2618,7 @@ def update(
             "interactive": interactive,
             "visible": visible,
             "value": value,
+            "autoplay": autoplay,
             "__type__": "update",
         }
 


Test patch:
diff --git a/test/test_blocks.py b/test/test_blocks.py
--- a/test/test_blocks.py
+++ b/test/test_blocks.py
@@ -1154,6 +1154,7 @@ def test_with_generic_update(self):
             }
         )
         assert specific_update == {
+            "autoplay": None,
             "source": None,
             "label": None,
             "show_label": None,
diff --git a/test/test_components.py b/test/test_components.py
--- a/test/test_components.py
+++ b/test/test_components.py
@@ -822,6 +822,7 @@ def test_component_functions(self):
 
         audio_input = gr.Audio(label="Upload Your Audio")
         assert audio_input.get_config() == {
+            "autoplay": False,
             "source": "upload",
             "name": "audio",
             "streaming": False,
@@ -856,6 +857,7 @@ def test_component_functions(self):
         audio_output = gr.Audio(type="filepath")
         assert filecmp.cmp(y_audio.name, audio_output.postprocess(y_audio.name)["name"])
         assert audio_output.get_config() == {
+            "autoplay": False,
             "name": "audio",
             "streaming": False,
             "show_label": True,
@@ -1306,6 +1308,7 @@ def test_component_functions(self):
 
         video_input = gr.Video(label="Upload Your Video")
         assert video_input.get_config() == {
+            "autoplay": False,
             "source": "upload",
             "name": "video",
             "show_label": True,