EdgeInstant AudioIn / Thinker / Talker

This model combines the Qwen3-ASR audio encoder, trained AudioIn projection and Qwen3.5 Thinker, compact native-token Talker, speaker conditions, and Qwen3-TTS waveform decoder. All weights, token mappings and input processors are included. Inference uses one Transformers 5.12.1 environment.

Install the dependencies in requirements.txt. The package uses custom modeling code, so load it with trust_remote_code=True.

import soundfile as sf
from transformers import AutoModel, AutoProcessor

path = "/path/to/EdgeInstant-1.5b-hf"
processor = AutoProcessor.from_pretrained(path, trust_remote_code=True)
model = AutoModel.from_pretrained(
    path, trust_remote_code=True, dtype="auto", device_map="cuda:0",
).eval()

waveform, sample_rate = sf.read("question.wav", dtype="float32")
inputs = processor(audio=waveform, sampling_rate=sample_rate).to("cuda:0")
reply = model.generate_speech(**inputs, max_new_tokens=256, do_sample=False)
print(processor.decode(reply["text_token_ids"], skip_special_tokens=True))
sf.write("reply.wav", reply["audio"].cpu().numpy(), reply["sampling_rate"])

Input audio must be mono, 16 kHz and at most 30 seconds. Output audio is 24 kHz. The bundled speaker is fixed. Keep the loaded dtype: the AudioIn projector and special-token deltas use FP32; the other weights use BF16.

Text-only input uses processor(text="你好"). For transcription, pass task="asr" to the processor and call model.generate(**inputs); generated sequences include the prompt, following the Hugging Face causal LM convention. AutoModelForCausalLM loads the same class.

model.synthesize(token_ids, language="chinese") speaks a sequence of native Thinker tokens. language also accepts "english" and "auto".

inputs = processor(text="请用一句话介绍你自己。").to("cuda:0")
for event in model.stream_generate(**inputs):
    if event["type"] == "text":
        print(processor.decode([event["token_id"]]), end="", flush=True)
    elif event["type"] == "audio":
        # Consecutive float32 waveform chunks, ready for a 24 kHz audio sink.
        audio_chunk = event["audio"].numpy()

Speech and streaming generation accept one conversation at a time. Text generation and audio-conditioned forward support batches. forward(..., labels=...) returns the Thinker language-model loss. Acoustic training remains in the repository's Talker training scripts. Streaming uses greedy Thinker decoding and the native Talker state machine, with configurable packet size and decoder context.

The model contains Qwen components and weights; their respective licenses apply. The adapted Qwen3-TTS decoder code is covered by LICENSE.codec.