RuiRuihigh/qwen2audioinstruct-sft-merged

Qwen2-Audio-7B-Instruct Emotional Dialogue SFT

This is a full merged SFT model based on Qwen/Qwen2-Audio-7B-Instruct.

It was fine-tuned for spoken-dialogue understanding, including generating a one-sentence emotional summary for each speaker in a conversation. The LoRA adapter has been merged into the base model weights, so no separate adapter is required for inference.

Intended use

Given a dialogue audio file, the model can follow an instruction such as:

Generate an emotional summary of each speaker throughout the conversation in one sentence. Use Person1 and Person2 to refer to the speakers.

Example output:

Person1 sounds frustrated but remains engaged, while Person2 is calm and reassuring throughout the conversation.

Usage

Install dependencies:

pip install -U transformers accelerate torch librosa soundfile

Run inference with a local audio file:

import torch
import librosa
import soundfile as sf
from transformers import AutoProcessor, Qwen2AudioForConditionalGeneration

# For a downloaded local model, replace this with your local directory.
model_id = "RuiRuihigh/qwen2audioinstruct-sft-merged"
audio_path = "dialogue.wav"

# Load processor and merged model.
processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = Qwen2AudioForConditionalGeneration.from_pretrained(
    model_id,
    trust_remote_code=True,
    device_map="auto",
    dtype=torch.float16,
).eval()

prompt = (
    "<|audio_bos|><|AUDIO|><|audio_eos|>"
    "Generate an emotional summary of each speaker throughout the conversation "
    "in one sentence. Use Person1 and Person2 to refer to the speakers."
)

# Load audio and convert stereo audio to mono if necessary.
audio, sr = sf.read(audio_path, always_2d=False)

if audio.ndim == 2:
    audio = audio.mean(axis=1)

audio = audio.astype("float32")

# Resample audio to the sampling rate required by Qwen2-Audio.
target_sr = processor.feature_extractor.sampling_rate
if sr != target_sr:
    audio = librosa.resample(
        audio,
        orig_sr=sr,
        target_sr=target_sr,
    )

inputs = processor(
    text=prompt,
    audio=audio,
    return_tensors="pt",
)

# Move every tensor input to the device hosting the model.
inputs = {
    key: value.to(model.device) if hasattr(value, "to") else value
    for key, value in inputs.items()
}

with torch.no_grad():
    generated_ids = model.generate(
        **inputs,
        max_new_tokens=256,
    )

# Remove prompt tokens and decode only the generated response.
generated_ids = generated_ids[:, inputs["input_ids"].size(1):]
response = processor.batch_decode(
    generated_ids,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
)[0]

print(response)

Notes

• This is a merged full model, not a LoRA adapter. Load the repository directly with from_pretrained.
• The model is intended for GPU inference. The example uses FP16 (dtype=torch.float16).
• Audio is converted to mono before resampling. This is important: a stereo array returned by soundfile has shape (samples, channels), and directly resampling it can otherwise resample the wrong axis.
• Person1 and Person2 labels are requested by the prompt; the model does not perform guaranteed speaker diarization.
• Outputs may be inaccurate and should not be used as the sole basis for high-stakes emotional or mental-health judgments.