Repository: SYSTRAN/faster-whisper
Issue category: Performance Issue

Issue:
OOM when using VAD
Hi, does somebody else experience issues with memory consumption when transcribing audio files containing a lot of speech (~ 4 hours long)? I am running the latest version of faster-whisper in a Kubernetes pod on a g4dn AWS instance. The server has 4 cores, 1 GPU, and 16GB RAM, but the pod is limited to 2 cores. The base image is `pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime` and as per [this pinned issue](https://github.com/SYSTRAN/faster-whisper/issues/1086) the installed versions should be compatible:
- python 3.11
- torch 2.5.1+cu124
- ctranslate2 4.5.0
- cuda 12.4
- cudnn 9.1.0.7

The process gets killed during the transcription phase when VAD is enabled. I tried the solution [described here](https://github.com/snakers4/silero-vad/issues/356), but it doesn't help. See the logs attached. Anyone has any idea what could be the cause of the OOM?

[libraries.txt](https://github.com/user-attachments/files/18039471/libraries.txt)
[logs on sigkill.txt](https://github.com/user-attachments/files/18039459/logs.on.sigkill.txt)


Hints:

Patch:
diff --git a/faster_whisper/vad.py b/faster_whisper/vad.py
index 9605931c..1f7d2057 100644
--- a/faster_whisper/vad.py
+++ b/faster_whisper/vad.py
@@ -260,8 +260,9 @@ def __init__(self, encoder_path, decoder_path):
             ) from e
 
         opts = onnxruntime.SessionOptions()
-        opts.inter_op_num_threads = 0
-        opts.intra_op_num_threads = 0
+        opts.inter_op_num_threads = 1
+        opts.intra_op_num_threads = 1
+        opts.enable_cpu_mem_arena = False
         opts.log_severity_level = 4
 
         self.encoder_session = onnxruntime.InferenceSession(
@@ -301,7 +302,16 @@ def __call__(
 
         batched_audio = batched_audio.reshape(-1, num_samples + context_size_samples)
 
-        encoder_output = self.encoder_session.run(None, {"input": batched_audio})[0]
+        encoder_batch_size = 10000
+        num_segments = batched_audio.shape[0]
+        encoder_outputs = []
+        for i in range(0, num_segments, encoder_batch_size):
+            encoder_output = self.encoder_session.run(
+                None, {"input": batched_audio[i : i + encoder_batch_size]}
+            )[0]
+            encoder_outputs.append(encoder_output)
+
+        encoder_output = np.concatenate(encoder_outputs, axis=0)
         encoder_output = encoder_output.reshape(batch_size, -1, 128)
 
         decoder_outputs = []


Test patch: