Repository: gradio-app/gradio
Pull request: Add sample rate config option to `gr.Audio()`
## Description

Closes: #6567

See Issue for background.
Note: Normally the sample rate should be the same as the source audio, but since I did not find an easy way to get the sampled audio of the source audio, set it to 44100Hz for now.

## 🎯 PRs Should Target Issues

Before your create a PR, please check to see if there is [an existing issue](https://github.com/gradio-app/gradio/issues) for this change. If not, please create an issue before you create this PR, unless the fix is very small. 

Not adhering to this guideline will result in the PR being closed. 

## Tests

1. PRs will only be merged if tests pass on CI. To run the tests locally, please set up [your Gradio environment locally](https://github.com/gradio-app/gradio/blob/main/CONTRIBUTING.md) and run the tests: `bash scripts/run_all_tests.sh`

2. You may need to run the linters: `bash scripts/format_backend.sh` and `bash scripts/format_frontend.sh`
  


Problem statement:
Trimming audio changes the sample rate
### Describe the bug

When using the trimming feature on an uploaded audio file, the sample rate is changed to 8 kHz for a 44.1 kHz file. 

### Have you searched existing issues?  🔎

- [X] I have searched and found no existing issues

### Reproduction

```
import librosa
import gradio as gr
def ui(launch_kwargs):
  x = gr.Audio(label="audio", type="filepath")
  ...
  button.click(run, inputs=[x])
def run(audio):
  x, sr = librosa.load(audio, mono=False, sr=None)
  print(sr)
```
When I upload a 44.1 kHz mp3 file and click the button, this function prints "44100"
After trimming and clicking the button, it prints "8000"


### Screenshot

_No response_

### Logs

_No response_

### System Info

```shell
Gradio Environment Information:
------------------------------
Operating System: Darwin
gradio version: 4.7.1
gradio_client version: 0.7.0

------------------------------------------------
gradio dependencies in your environment:

aiofiles: 23.2.1
altair: 5.1.2
fastapi: 0.104.1
ffmpy: 0.3.1
gradio-client==0.7.0 is not installed.
httpx: 0.25.1
huggingface-hub: 0.19.4
importlib-resources: 6.1.1
jinja2: 3.1.2
markupsafe: 2.1.3
matplotlib: 3.8.2
numpy: 1.26.2
orjson: 3.9.10
packaging: 23.2
pandas: 2.1.3
pillow: 10.1.0
pydantic: 2.5.1
pydub: 0.25.1
python-multipart: 0.0.6
pyyaml: 6.0.1
requests: 2.31.0
semantic-version: 2.10.0
tomlkit==0.12.0 is not installed.
typer: 0.9.0
typing-extensions: 4.8.0
uvicorn: 0.24.0.post1
authlib; extra == 'oauth' is not installed.
itsdangerous; extra == 'oauth' is not installed.


gradio_client dependencies in your environment:

fsspec: 2023.10.0
httpx: 0.25.1
huggingface-hub: 0.19.4
packaging: 23.2
requests: 2.31.0
typing-extensions: 4.8.0
websockets: 11.0.3
```


### Severity

Blocking usage of gradio

Hints from issue discussion:
@hannahblair 
I have also encountered this strange problem. I am in the process of doing so, but will report the results of my investigation.
The audio trimming function is implemented on the JavaScript (Svelte) side by sourcing the AudioBuffer obtained from WaveSurfer.js.
However, if the sample rate is not specified when WaveSurfer.js is initialized, the AudioBuffer held internally by WaveSurfer.js will be 8000Hz (which has no effect on playback, which is annoying). By the way, the sample rate supported by browsers seems to be 8000Hz.
https://github.com/katspaugh/wavesurfer.js/blob/main/src/wavesurfer.ts#L62
Specifying the correct sample rate when initializing WaveSurfer.js should fix this problem.
Thanks so much for the extra context @tsukumijima! That's really helpful. 

We welcome contributor PRs if you're interested, but otherwise I'll look into this ASAP. 😊 
@hannahblair 
Thanks for the reply.
I thought about writing a PR, but I'm having trouble figuring out how to get the sample rate of the source beforehand and how to implement it from an internal design standpoint, so I'll leave the fix to you.

In my environment, as a workaround, I have forcibly rewritten Gradio's JS dist with the following code and confirmed that it can be saved as 44100 Hz.

```python
    index_js_path = Path(gradio.__file__).parent / 'templates/frontend/assets/index-84ec2915.js'
    with open(index_js_path, 'r') as f:
        index_js = f.read()
    index_js = index_js.replace('sampleRate: 8000,', 'sampleRate: 44100,')
    with open(index_js_path, 'w') as f:
        f.write(index_js)
```

In addition, a newly discovered problem seems to be that audio that is about to be cracked may experience clipping (a momentary unpleasant noise) after trimming.

There may be a problem with the implementation of the trimming process or the WAV export process.
For your reference, I have attached a wav file that crackles when trimming (after being able to save at 44100Hz).
[0001_この電車は中央線快速八王子行きです。.wav.zip](https://github.com/gradio-app/gradio/files/13659854/0001_.wav.zip)


As for clipping, it seems to be caused by not respecting the volume of the original audio and raising the volume above the original.
By respecting the volume of the original audio and trimming it, audio clipping (unpleasant noise) can be avoided.
@hannahblair 
Upon further investigation, it seems that the problem is caused by WaveSurfer.js normalizing the audio when it loads it.
Including the sample rate issue, I think it would be better to include a process to decode the audio directly on the Gradio side to get the audio buffer, instead of getting the audio buffer from WaveSurfer.js.
https://github.com/katspaugh/wavesurfer.js/blob/main/src/decoder.ts#L8-L25
Upon further investigation, I found that fetching data directly with fetch() and retrieving the AudioBuffer from decodeAudioData() also causes clipping, as does retrieving the AudioBuffer from WaveSurfer.js.
This may be a problem with the Web Audio API itself.
@hannahblair 
In the end, the clipping was caused by improper processing of PCM data write in audioBufferToWav().
Created pull request. I'm not sure if this is good code for Gradio, so I'll leave it up to you to decide if you want to merge it as is or make another revised commit for reference.
https://github.com/gradio-app/gradio/pull/6826 
Thanks so much for addressing this issue @tsukumijima! I'll review it as soon as I can.

Commit message: Merge branch 'main' into main

Patch to review:
diff --git a/.changeset/weak-streets-check.md b/.changeset/weak-streets-check.md
new file mode 100644
index 0000000000..991d94d657
--- /dev/null
+++ b/.changeset/weak-streets-check.md
@@ -0,0 +1,6 @@
+---
+"@gradio/audio": patch
+"gradio": patch
+---
+
+fix:Fix a bug that caused the sample rate of audio to be 8000 Hz after trimming and a bug that caused volume amplification and clipping each time trimming was performed
diff --git a/js/audio/Index.svelte b/js/audio/Index.svelte
index a34278e524..917bc5a3e2 100644
--- a/js/audio/Index.svelte
+++ b/js/audio/Index.svelte
@@ -104,7 +104,8 @@
 		dragToSeek: true,
 		normalize: true,
 		minPxPerSec: 20,
-		mediaControls: waveform_options.show_controls
+		mediaControls: waveform_options.show_controls,
+		sampleRate: 44100
 	};
 
 	const trim_region_settings = {
diff --git a/js/audio/recorder/AudioRecorder.svelte b/js/audio/recorder/AudioRecorder.svelte
index aedc24c419..6b27d3ee77 100644
--- a/js/audio/recorder/AudioRecorder.svelte
+++ b/js/audio/recorder/AudioRecorder.svelte
@@ -82,7 +82,7 @@
 		timing = false;
 		clearInterval(interval);
 		const array_buffer = await blob.arrayBuffer();
-		const context = new AudioContext();
+		const context = new AudioContext({ sampleRate: 44100 });
 		const audio_buffer = await context.decodeAudioData(array_buffer);
 
 		if (audio_buffer)
diff --git a/js/audio/shared/audioBufferToWav.ts b/js/audio/shared/audioBufferToWav.ts
index 83872e6c0f..578821cf04 100644
--- a/js/audio/shared/audioBufferToWav.ts
+++ b/js/audio/shared/audioBufferToWav.ts
@@ -47,7 +47,9 @@ export function audioBufferToWav(audioBuffer: AudioBuffer): Uint8Array {
 	for (let i = 0; i < audioBuffer.numberOfChannels; i++) {
 		const channel = audioBuffer.getChannelData(i);
 		for (let j = 0; j < channel.length; j++) {
-			view.setInt16(offset, channel[j] * 0xffff, true);
+			// Scaling Float32 to Int16
+			const sample = Math.max(-1, Math.min(1, channel[j]));
+			view.setInt16(offset, sample * 0x7fff, true);
 			offset += 2;
 		}
 	}
diff --git a/js/audio/shared/utils.ts b/js/audio/shared/utils.ts
index 033949766a..663074ffb2 100644
--- a/js/audio/shared/utils.ts
+++ b/js/audio/shared/utils.ts
@@ -20,7 +20,7 @@ export const process_audio = async (
 	start?: number,
 	end?: number
 ): Promise<Uint8Array> => {
-	const audioContext = new AudioContext();
+	const audioContext = new AudioContext({ sampleRate: audioBuffer.sampleRate });
 	const numberOfChannels = audioBuffer.numberOfChannels;
 	const sampleRate = audioBuffer.sampleRate;
 


Code review comments:
diff_hunk: @@ -104,7 +104,8 @@
 		dragToSeek: true,
 		normalize: true,
 		minPxPerSec: 20,
-		mediaControls: waveform_options.show_controls
+		mediaControls: waveform_options.show_controls,
+		sampleRate: 44100
path: js/audio/Index.svelte
text: @user1:
Just a thought - it would be great to make this a customisable option in `waveform_options` (and set default to 44100)

@author:
I agree with the idea too.
I was going to use this trimming feature for annotation work on a TTS dataset, but I made this change because the voice in the dataset must be 44100Hz by specification.
Realistically, given the use case, I think it would be fine if the sample rate could be specified in waveform_options.

However, I am not familiar with Gradio's internals and am not confident that I can make the changes correctly. 
I would like to leave any further changes to you.