TASTE2-8B-EN-Instruct-dircect-FT

Part of the TASTE2 project — Text-Aligned Speech Modeling and Deployment toward Full-Duplex Voice Interaction.

• 📄 Paper: TASTE2 paper (PDF)
• 🌐 Project homepage: https://gitycc.github.io/TASTE2-Homepage/
• 💻 Spoken LM code: https://github.com/GitYCC/TASTE-SpokenLM-2
• 💻 VoiceBot code: https://github.com/GitYCC/TASTE-Voice-Bot
• 🤗 All TASTE2 checkpoints: https://huggingface.co/collections/YC-Chen/taste2
• 📚 Dataset: https://huggingface.co/datasets/wilzzzz/paralinguistic_dialogues

About TASTE2

Full-duplex voice interaction requires more than converting one complete utterance into another. A system must process speech as it arrives, decide when to take or yield the floor, and stop when the user interrupts, while preserving pretrained linguistic competence and acoustic paralinguistic cues. TASTE2 turns the original utterance-level TASTE method into an incremental dialogue stack of three components — a Speech Tokenizer, a Spoken LM, and a Speech Detokenizer — aligning one continuous audio latent with every text token so the token sequence length is preserved end-to-end.

A shared text-token vocabulary removes word-level averaging and language-dependent segmentation, modality-aligned dialogue training predicts one continuous audio latent per text token without interleaving heterogeneous token streams, and an incremental Speech Detokenizer streams synthesis through CosyVoice2. On top of this, TASTE2 VoiceBot is a working research system that processes user speech incrementally, streams synthesized audio, and stops generation when the user barges in.

This checkpoint

TASTE2-8B-EN-Instruct-dircect-FT starts from the base TASTE2-8B-EN-Instruct checkpoint and applies direct fine-tuning (referred to as direct-ft), one of two fine-tuning strategies explored for the Instruct model — the other being TASTE2-8B-EN-Instruct-merge-FT (merge-ft).

Files

This repo bundles the full TASTE2 inference stack for this checkpoint: Spoken LM weights (llm.pt, slm.pt, plus intermediate references slm_origin.pt/slm_IF.pt), CosyVoice2-based flow/vocoder weights (flow.pt, flow.cache.pt, hift.pt), the speech tokenizer (speech_tokenizer_v2.onnx, campplus.onnx), the bundled qwen2.5-7b-instruct/ and CosyVoice-BlankEN/ base checkpoints, distil-whisper/, and training config yamls.

Not included: flow.decoder.estimator.fp32.mygpu.plan and the cosyvoice-trtllm/ engine directory. These are TensorRT engines compiled for a specific GPU/driver combination and are not portable — rebuild them locally with the CosyVoice2/TensorRT-LLM export scripts if you need accelerated inference.

For setup and inference instructions, see the TASTE-SpokenLM-2 and TASTE-Voice-Bot repositories.

Authors

Yi-Chang Chen*¹, Chun Wei Chen*^2,3, Dien-Ruei Wu*^2,3, Jie Lin³, Hung-yi Lee³, Da-Shan Shiu¹

¹ MediaTek Research · ² Internship at MediaTek Research · ³ National Taiwan University · *Equal contribution

Citation

BibTeX will be added once the public paper record is available. In the meantime, please refer to the project homepage and the paper PDF.

License / Disclaimer

This checkpoint is provided for academic and research purposes only, © 2026 MediaTek Research and National Taiwan University. It is built on top of Qwen2.5-7B-Instruct and CosyVoice2, which retain their own respective upstream licenses — please review those separately before use.