error : ubuntu@ubuntu-virtual-machine:~/Desktop/new$ ./main.py
/home/ubuntu/.local/lib/python3.10/site-packages/pyannote/audio/core/io.py:43: UserWarning: torchaudio._backend.set_audio_backend has been deprecated. With dispatcher enabled, this function is no-op. You can remove the function call.
torchaudio.set_audio_backend("soundfile")
/home/ubuntu/.local/lib/python3.10/site-packages/pyannote/audio/pipelines/speaker_verification.py:43: UserWarning: torchaudio._backend.get_audio_backend has been deprecated. With dispatcher enabled, this function is no-op. You can remove the function call.
backend = torchaudio.get_audio_backend()
INFO:speechbrain.utils.quirks:Applied quirks (see speechbrain.utils.quirks): [allow_tf32, disable_jit_profiling]
INFO:speechbrain.utils.quirks:Excluded quirks specified by the SB_DISABLE_QUIRKS environment (comma-separated list): []
/home/ubuntu/.local/lib/python3.10/site-packages/pyannote/audio/pipelines/speaker_verification.py:45: UserWarning: Module 'speechbrain.pretrained' was deprecated, redirecting to 'speechbrain.inference'. Please update your script. This is a change from SpeechBrain 1.0. See: https://github.com/speechbrain/speechbrain/releases/tag/v1.0.0
from speechbrain.pretrained import (
/home/ubuntu/.local/lib/python3.10/site-packages/pyannote/audio/pipelines/speaker_verification.py:53: UserWarning: torchaudio._backend.set_audio_backend has been deprecated. With dispatcher enabled, this function is no-op. You can remove the function call.
torchaudio.set_audio_backend(backend)
/home/ubuntu/.local/lib/python3.10/site-packages/pyannote/audio/tasks/segmentation/mixins.py:37: UserWarning: torchaudio.backend.common.AudioMetaData has been moved to torchaudio.AudioMetaData. Please update the import path.
from torchaudio.backend.common import AudioMetaData
Initializing AudioProcessor with cpu
Loading Whisper v2 model: base
No language specified, language will be first be detected for each audio file (increases inference time).
Lightning automatically upgraded your loaded checkpoint from v1.5.4 to v2.4.0. To apply the upgrade to your files permanently, run python -m pytorch_lightning.utilities.upgrade_checkpoint ../../.cache/torch/whisperx-vad-segmentation.bin
Model was trained with pyannote.audio 0.0.1, yours is 3.1.1. Bad things might happen unless you revert pyannote.audio to 0.x.
Model was trained with torch 1.10.0+cu102, yours is 2.5.1. Bad things might happen unless you revert torch to 1.x.
Loading Demucs model...
Loading VAD model...
Downloading: "https://github.com/snakers4/silero-vad/zipball/master" to /home/ubuntu/.cache/torch/hub/master.zip
Models loaded successfully!
=== Extracting Vocals Debug Information ===
Input audio path: /home/ubuntu/Downloads/audio.mp3

    Loading audio file...

    Audio loaded successfully
    Audio shape: torch.Size([2, 599040])
    Sample rate: 48000
    Audio moved to device: cpu
    Chunk duration: 30 seconds
    Chunk samples: 1440000
    Total audio samples: 599040

    Processing full audio...

    Audio shape after unsqueeze: torch.Size([1, 2, 599040])
    apply_model result:
        Type of sources: <class 'torch.Tensor'>
        Sources shape: torch.Size([1, 4, 2, 599040])
    Vocals shape: torch.Size([2, 599040])
    Audio shape before transpose: (2, 599040)
    Transposed audio shape: (599040, 2)
    Audio min: -0.6916637420654297, max: 0.8381609320640564, mean: -0.00017020900850184262
    Audio data written to /home/ubuntu/Desktop/new/temp/temp_in.wav
    Read test audio from file with shape (599040, 2) and sample rate 48000
    ffmpeg stdout:
    ffmpeg stderr: ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers
    built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
    configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1 --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
    libavutil 56. 70.100 / 56. 70.100
    libavcodec 58.134.100 / 58.134.100
    libavformat 58. 76.100 / 58. 76.100
    libavdevice 58. 13.100 / 58. 13.100
    libavfilter 7.110.100 / 7.110.100
    libswscale 5. 9.100 / 5. 9.100
    libswresample 3. 9.100 / 3. 9.100
    libpostproc 55. 9.100 / 55. 9.100
    Guessed Channel Layout for Input Stream #0.0 : stereo
    Input #0, wav, from '/home/ubuntu/Desktop/new/temp/temp_in.wav':
    Duration: 00:00:12.48, bitrate: 1536 kb/s
    Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 48000 Hz, stereo, s16, 1536 kb/s
    Stream mapping:
    Stream #0:0 -> #0:0 (pcm_s16le (native) -> pcm_s16le (native))
    Press [q] to stop, [?] for help
    Output #0, wav, to '/home/ubuntu/Desktop/new/temp/temp_out.wav':
    Metadata:
    ISFT : Lavf58.76.100
    Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 48000 Hz, stereo, s16, 1536 kb/s
    Metadata:
    encoder : Lavc58.134.100 pcm_s16le
    size= 0kB time=00:00:00.00 bitrate=N/A speed=N/A
    size= 2304kB time=00:00:09.57 bitrate=1970.3kbits/s speed=17.4x
    size= 2340kB time=00:00:12.47 bitrate=1536.1kbits/s speed=22.7x
    video:0kB audio:2340kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.003255%

Applying VAD...
Adding markers...
Applying compression...
Transcribing...
Transcription error: FasterWhisperPipeline.transcribe() got an unexpected keyword argument 'beam_size'
Transcribing...
Transcription error: FasterWhisperPipeline.transcribe() got an unexpected keyword argument 'beam_size'
Transcribing...
Transcription error: FasterWhisperPipeline.transcribe() got an unexpected keyword argument 'beam_size'

script :

import whisperx
import torch
import torchaudio
from demucs.pretrained import get_model
from demucs.apply import apply_model
import numpy as np
import soundfile as sf
import subprocess
from pathlib import Path
import os
from config import *
from torch.hub import load as torch_hub_load
from silero_vad import load_silero_vad, get_speech_timestamps

class AudioProcessor:
def init(self, model_size=DEFAULT_MODEL_SIZE, whisper_version=DEFAULT_WHISPER_VERSION):
torch.set_num_threads(CPU_THREADS)
print(f"Initializing AudioProcessor with {DEVICE}")
self.init_models(model_size, whisper_version)

def init_models(self, model_size, whisper_version):
    try:
        print(f"Loading Whisper {whisper_version} model: {model_size}")
        self.whisper_model = whisperx.load_model(
            model_size,
            device=DEVICE,
            compute_type="float32",
            download_root=str(MODELS_DIR),
            language=None,        # Set to None for automatic language detection, or specify 'en' for English
        )

        print("Loading Demucs model...")
        self.demucs_model = get_model(name="htdemucs_ft")
        self.demucs_model.to(DEVICE)

        print("Loading VAD model...")
        # Correct way to load Silero VAD
        self.vad_model, utils = torch_hub_load(
            repo_or_dir='snakers4/silero-vad',
            model='silero_vad',
            force_reload=True
        )

        # Unpack utils properly
        (self.get_speech_timestamps,
            self.save_audio,
            self.read_audio,
            self.VADIterator,
            self.collect_chunks) = utils

        self.vad_model = self.vad_model.to(DEVICE)
        print("Models loaded successfully!")

    except Exception as e:
        print(f"Error initializing models: {e}")
        raise

def process_file(self, input_file, output_file, beam_size=2, patience=1, temperature=0.0):
    """Main processing pipeline"""
    try:
        # 1. Extract vocals
        vocals, sr = self.extract_vocals(input_file)

        # 2. Normalize and remove silence
        normalized = self.normalize_audio(vocals, sr)

        # 3. Apply VAD
        vad_audio = self.apply_vad(normalized)

        # 4. Add markers
        marked_audio = self.add_markers(vad_audio)

        # 5. Apply compression
        compressed = self.apply_compression(marked_audio)

        # Try transcription with different approaches
        transcription = None

        # With markers
        transcription = self.transcribe(compressed, beam_size, patience, temperature)
        if self.verify_transcription(transcription):
            return self.save_transcription(transcription, output_file)

        # With inverted markers
        transcription = self.transcribe(-compressed, beam_size, patience, temperature)
        if self.verify_transcription(transcription):
            return self.save_transcription(transcription, output_file)

        # Without markers
        transcription = self.transcribe(vad_audio, beam_size, patience, temperature)
        return self.save_transcription(transcription, output_file)

    except Exception as e:
        print(f"Error processing file: {e}")
        raise
    finally:
        # Implement cleanup_temp if necessary
        pass  # Replace this with cleanup_temp() if you have that function

def extract_vocals(self, audio_path):
    """Extract vocals using Demucs"""
    print("=== Extracting Vocals Debug Information ===")
    print(f"Input audio path: {audio_path}")

    try:
        # Load audio
        print("1. Loading audio file...")
        audio, sr = torchaudio.load(audio_path)
        print(f"- Audio loaded successfully")
        print(f"- Audio shape: {audio.shape}")
        print(f"- Sample rate: {sr}")

        # Move to device
        audio = audio.to(DEVICE)
        print(f"- Audio moved to device: {DEVICE}")

        # Chunk processing setup
        chunk_samples = int(CHUNK_DURATION * sr)
        print(f"- Chunk duration: {CHUNK_DURATION} seconds")
        print(f"- Chunk samples: {chunk_samples}")
        print(f"- Total audio samples: {audio.shape[1]}")

        # Source list in order
        sources_list = ['drums', 'bass', 'other', 'vocals']
        vocals_index = sources_list.index('vocals')  # Should be index 3

        # Process chunked or full audio
        if audio.shape[1] > chunk_samples:
            print("2. Processing audio in chunks...")
            vocals_chunks = []
            for i in range(0, audio.shape[1], chunk_samples):
                chunk = audio[:, i:min(i+chunk_samples, audio.shape[1])]
                print(f"  - Processing chunk {i//chunk_samples + 1}")
                print(f"  - Chunk shape before unsqueeze: {chunk.shape}")

                # Add batch dimension
                chunk = chunk.unsqueeze(0)  # Shape becomes (1, channels, length)
                print(f"  - Chunk shape after unsqueeze: {chunk.shape}")

                try:
                    # Apply model
                    with torch.no_grad():
                        sources = apply_model(self.demucs_model, chunk, device=DEVICE)
                    print("  - apply_model result:")
                    print(f"    - Type of sources: {type(sources)}")
                    print(f"    - Sources shape: {sources.shape}")

                    # Extract vocals
                    if isinstance(sources, torch.Tensor):
                        vocals_chunk = sources[:, vocals_index, :, :]
                        vocals_chunk = vocals_chunk.squeeze(0)
                    else:
                        raise ValueError("Unexpected sources type or structure")

                    if vocals_chunk is None:
                        raise ValueError("No vocals extracted from chunk")

                    print(f"    - Vocals chunk shape: {vocals_chunk.shape}")
                    vocals_chunks.append(vocals_chunk)

                except Exception as chunk_error:
                    print(f"  !! Error processing chunk {i//chunk_samples + 1}: {chunk_error}")
                    raise

            vocals = torch.cat(vocals_chunks, dim=1)
            print(f"- Concatenated vocals shape: {vocals.shape}")

        else:
            print("2. Processing full audio...")
            # Add batch dimension
            audio = audio.unsqueeze(0)  # Shape becomes (1, channels, length)
            print(f"  - Audio shape after unsqueeze: {audio.shape}")

            # Apply model
            with torch.no_grad():
                sources = apply_model(self.demucs_model, audio, device=DEVICE)
            print("  - apply_model result:")
            print(f"    - Type of sources: {type(sources)}")
            print(f"    - Sources shape: {sources.shape}")

            # Extract vocals
            if isinstance(sources, torch.Tensor):
                vocals = sources[:, vocals_index, :, :]
                vocals = vocals.squeeze(0)
            else:
                raise ValueError("Unexpected sources type or structure")

            if vocals is None:
                raise ValueError("No vocals extracted from full audio")

            print(f"  - Vocals shape: {vocals.shape}")

        return vocals, sr

    except Exception as e:
        print("=== FATAL ERROR IN EXTRACT_VOCALS ===")
        print(f"Error details: {e}")
        import traceback
        traceback.print_exc()
        raise

def normalize_audio(self, audio, sr):
    """Normalize audio and remove silence using ffmpeg"""
    import soundfile as sf
    temp_in = TEMP_DIR / "temp_in.wav"
    temp_out = TEMP_DIR / "temp_out.wav"

    try:
        # Ensure TEMP_DIR exists
        TEMP_DIR.mkdir(parents=True, exist_ok=True)
        
        # Convert audio tensor to NumPy array
        audio_np = audio.cpu().numpy()
        print(f"Audio shape before transpose: {audio_np.shape}")

        # Transpose if necessary
        if audio_np.ndim == 2:
            audio_np = audio_np.T  # Shape becomes (samples, channels)
            print(f"Transposed audio shape: {audio_np.shape}")

        # Ensure data type is float32
        if audio_np.dtype != np.float32:
            audio_np = audio_np.astype(np.float32)

        # Check for NaNs or Infs
        if np.isnan(audio_np).any() or np.isinf(audio_np).any():
            raise ValueError("Audio data contains NaNs or Infs.")

        # Print audio stats
        print(f"Audio min: {audio_np.min()}, max: {audio_np.max()}, mean: {audio_np.mean()}")

        # Write audio data to temp_in.wav
        sf.write(str(temp_in), audio_np, sr, format='WAV', subtype='PCM_16')
        print(f"Audio data written to {temp_in}")

        # Test if the file can be read
        test_audio, test_sr = sf.read(str(temp_in))
        print(f"Read test audio from file with shape {test_audio.shape} and sample rate {test_sr}")

        # Run ffmpeg to normalize and remove silence
        cmd = [
            'ffmpeg', '-y',
            '-i', str(temp_in),
            '-af', (
                f"silenceremove=stop_periods=-1:stop_duration=1:stop_threshold={VAD_THRESHOLD}dB,"
                "loudnorm=I=-16:TP=-1.5:LRA=11"
            ),
            '-ar', str(sr),
            str(temp_out)
        ]

        process = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
        print(f"ffmpeg stdout: {process.stdout}")
        print(f"ffmpeg stderr: {process.stderr}")

        if process.returncode != 0:
            raise Exception(f"ffmpeg error: {process.stderr}")

        normalized, sr = torchaudio.load(str(temp_out))
        normalized = normalized.squeeze(0)
        return normalized

    except Exception as e:
        print(f"Error normalizing audio: {e}")
        raise

def apply_vad(self, audio):
    """Apply Voice Activity Detection"""
    print("Applying VAD...")
    audio_mono = audio.mean(dim=0) if audio.dim() > 1 else audio
    audio_numpy = audio_mono.cpu().numpy()

    # Process in chunks
    chunk_samples = int(CHUNK_DURATION * SAMPLE_RATE)
    speech_timestamps = []

    for i in range(0, len(audio_numpy), chunk_samples):
        chunk = audio_numpy[i:min(i + chunk_samples, len(audio_numpy))]
        chunk_tensor = torch.FloatTensor(chunk).to(DEVICE)

        # Get speech timestamps for the chunk
        chunk_timestamps = self.get_speech_timestamps(
            chunk_tensor,
            self.vad_model,
            sampling_rate=SAMPLE_RATE
        )

        # Adjust timestamps for chunk position
        for ts in chunk_timestamps:
            ts['start'] += i
            ts['end'] += i
        speech_timestamps.extend(chunk_timestamps)

    # Create masked audio
    masked_audio = torch.zeros_like(audio_mono)
    for ts in speech_timestamps:
        start_frame = ts['start']
        end_frame = ts['end']
        if start_frame < len(masked_audio) and end_frame <= len(masked_audio):
            masked_audio[start_frame:end_frame] = audio_mono[start_frame:end_frame]

    return masked_audio

def add_markers(self, audio):
    """Add voice markers at speech boundaries"""
    print("Adding markers...")
    marked_audio = audio.clone()

    # Generate marker signal
    t = np.arange(int(MARKER_DURATION * SAMPLE_RATE))
    marker = torch.from_numpy(
        0.1 * np.sin(2 * np.pi * MARKER_FREQUENCY * t / SAMPLE_RATE)
    ).float()

    # Find speech segments
    speech_segments = self.find_speech_segments(audio)

    # Add markers at segment boundaries
    for start, end in speech_segments:
        if start + len(marker) < len(marked_audio):
            marked_audio[start:start + len(marker)] += marker
        if end - len(marker) >= 0:
            marked_audio[end - len(marker):end] += marker

    return marked_audio

def find_speech_segments(self, audio, threshold=0.01):
    """Find speech segments in audio"""
    is_speech = torch.abs(audio) > threshold
    changes = torch.diff(is_speech.long())
    change_points = torch.nonzero(changes).flatten()

    segments = []
    for i in range(0, len(change_points)-1, 2):
        start = change_points[i].item()
        end = change_points[i+1].item() if i+1 < len(change_points) else len(audio)-1
        segments.append((start, end))

    return segments

def apply_compression(self, audio):
    """Apply speech compression using ffmpeg"""
    print("Applying compression...")
    temp_in = TEMP_DIR / "temp_compress_in.wav"
    temp_out = TEMP_DIR / "temp_compress_out.wav"

    try:
        sf.write(str(temp_in), audio.cpu().numpy(), SAMPLE_RATE)

        cmd = [
            'ffmpeg', '-y',
            '-i', str(temp_in),
            '-af', 'acompressor=threshold=-12dB:ratio=2:attack=200:release=1000',
            '-ar', str(SAMPLE_RATE),
            str(temp_out)
        ]

        subprocess.run(cmd, check=True, capture_output=True)
        compressed, _ = torchaudio.load(str(temp_out))
        return compressed.squeeze()

    except Exception as e:
        print(f"Error applying compression: {e}")
        raise

def transcribe(self, audio, beam_size=2, patience=1, temperature=0.0):
    """Transcribe audio using WhisperX"""
    print("Transcribing...")
    temp_file = TEMP_DIR / "temp_transcribe.wav"

    try:
        sf.write(str(temp_file), audio.cpu().numpy(), SAMPLE_RATE)

        # Load and transcribe
        audio_data = whisperx.load_audio(str(temp_file))
        result = self.whisper_model.transcribe(
            audio_data,
            batch_size=1,  # Optimized for CPU
            beam_size=beam_size,
            patience=patience,
            temperature=temperature,
        )

        # Align timestamps
        model_a, metadata = whisperx.load_align_model(
            language_code=result["language"],
            device=DEVICE
        )

        result = whisperx.align(
            result["segments"],
            model_a,
            metadata,
            audio_data,
            DEVICE,
            extend_duration=0.0
        )

        return result

    except Exception as e:
        print(f"Transcription error: {e}")
        return None

def verify_transcription(self, transcription):
    """Verify if transcription contains markers"""
    if not transcription or 'segments' not in transcription:
        return False

    marker_patterns = ["*", "beep", "tone", "[noise]", "[sound]"]
    return any(
        any(pattern in segment['text'].lower() for pattern in marker_patterns)
        for segment in transcription['segments']
    )

def save_transcription(self, transcription, output_file):
    """Save transcription to file"""
    if not transcription:
        return False

    try:
        with open(output_file, 'w', encoding='utf-8') as f:
            for segment in transcription['segments']:
                f.write(
                    f"[{segment['start']:.2f} - {segment['end']:.2f}] {segment['text']}\n"
                )
        return True
    except Exception as e:
        print(f"Error saving transcription: {e}")
        return False