DuplexJev-4B-Para (v1.3)

📄 Paper: arXiv:2610.02638 · Code: github.com/adventists-ai/duplexjev · Release notes: Japanese as a third language · weak languages

The small DuplexJev that hears the most beyond the words, now with Japanese close to its Chinese and English.

v1.3 (2026-10-22) replaces v1.2 (2026-10-21). Earlier versions stay available as revisions v1.2, v1.1 and v1.0 (--revision v1.2 in vLLM, revision="v1.2" in from_pretrained).

v1.2 brought Japanese from 60 % to 36 % error; v1.3 halves it again (FLEURS 17.2 %, conversational Japanese 13.1 %), with Chinese and English unchanged.

Architecture: Qwen3-ASR-0.6B encoder + connector + Qwen3-4B, 4.2 B parameters, bf16, ~10 GB GPU memory.

What changed

Continued from v1.2 in two stages (connector + a rank-64 LoRA on the LLM each time, merged here; 4,000 steps each), the full v1.2 mix replayed, plus:

added data	items	what it teaches
Japanese speech: Emilia JA, Emilia-YODAS JA, Common Voice ja and other Japanese speech data	358 k	transcribe; every label re-checked against Qwen3-ASR-0.6B, disagreeing clips dropped
Japanese continuation (text teacher: Qwen3-32B)	20 k	understand, not just repeat
Mandarin conversational paralinguistics, second version (licensed MagicData corpus, training speakers only)	42 k	adds stutter / repeated word, which character is drawn out, sudden noise, unnatural pause

Japanese was weighted 15,000 in the first stage and 30,000 in the second.

Results

Japanese (lower is better; FLEURS = median per-utterance character error on the test split; Emilia = a held-out Emilia JA shard of conversational speech; CV = Common Voice ja test; 150 utterances each):

	FLEURS ja	Emilia ja	Common Voice ja
v1.1	59.6	60.0	62.5
v1.2	35.8	27.5	39.7
v1.3	17.2	13.1	20.3
encoder ceiling (Qwen3-ASR-0.6B)	7.0	8.3	10.8

FLEURS test, other languages (median word / character error):

	ko	vi	th	ar	ru	zh	en
v1.2	20.3	41.3	33.3	58.6	36.4	9.4	5.6
v1.3	18.8	38.4	33.0	57.1	34.5	8.2	5.3

Serve with vLLM

pip install "vllm[audio]>=0.29" duplexjev-vllm
vllm serve adventists-ai/DuplexJev-4B-Para --max-model-len 4096
# previous version: add --revision v1.2

Decision prompts are the same as for DuplexJev-4B-Turn; see examples/vllm_client.py. Always offer a "none" option for stress and non-verbal questions. For conversation, send the user audio as the user turn (<|audio|>) with ordinary chat history and sample as usual.

Limitations

• Japanese is now close to Chinese and English on read and conversational speech, but still about twice the encoder's own error. Korean, Vietnamese, Thai and Arabic are about where v1.2 left them; the 32B model is much stronger there.
• Asked about itself, it may describe abilities (seeing images, speaking) that this checkpoint does not have on its own: it takes speech and text and replies in text.
• Speaker judgments are coarse and must not be used to identify or verify people; emotion outputs must not be used to make decisions about individuals.

License

CC BY-NC 4.0, non-commercial use only.

• Non-commercial data:
  • Stress: DEBATE (github.com/SmileHnu/DEBATE), Stress-17K and TinyStress-15K (slprl);
  • Expresso (Meta), EARS (Meta) and NonVerbalSpeech-38K, all CC BY-NC 4.0;
  • ESD (research only); SmoothConv (CC BY-NC).
  • Conversation replay: user turns from wilzzzz/paralinguistic_dialogues (no licence stated on the dataset card; used for non-commercial research), SmoothConv (CC BY-NC), AISHELL-1 (Apache-2.0), LibriSpeech (CC BY 4.0); target replies generated by Qwen3-4B (Apache-2.0); spoken chat questions synthesised with CosyVoice2 (Apache-2.0).
• Added in v1.2 (non-commercial or share-alike): Emilia JA/KO (CC BY-NC 4.0); VIVOS (CC BY-NC-SA 4.0); a licensed Mandarin conversational corpus with paralinguistic annotation (MagicData, purchased licence; audio not redistributed).
• Added in v1.2 (other): Emilia-YODAS JA/KO (CC BY 4.0), YODAS2 (CC BY 3.0), Zeroth-Korean (CC BY 4.0), FLEURS train split (CC BY 4.0; the test split was never used), Common Voice 17 ar/ru (CC0). Labels were re-checked, and for unlabelled YODAS2 clips produced, with Qwen3-ASR-0.6B (Apache-2.0); continuation targets were written by Qwen3-32B (Apache-2.0).
• Added in v1.3: Japanese speech from Emilia JA (CC BY-NC 4.0), Emilia-YODAS JA (CC BY 4.0), Common Voice ja (CC0) and other Japanese speech data, used for non-commercial research; every label re-checked with Qwen3-ASR-0.6B (Apache-2.0), continuation targets written by Qwen3-32B (Apache-2.0). The held-out Emilia JA shard used for testing was never trained on.
• Other data:
  • VocalSound (CC BY-SA 4.0), NonverbalTTS (audio from VoxCeleb and Expresso);
  • VoxCeleb1 (CC BY 4.0), AISHELL-1 (Apache-2.0), LibriSpeech (CC BY 4.0);
  • ARC (CC BY-SA 4.0), OpenBookQA, CommonsenseQA (MIT).
• Base models keep their licences: Qwen3-4B and Qwen3-ASR-0.6B are Apache-2.0 (Alibaba Cloud); the Qwen3-4B licence is included. The LLM weights are Qwen3-4B with LoRAs merged in.

Citation

@misc{jin2026duplexjev,
  title         = {Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen {LLM} Hear Beyond the Transcript},
  author        = {Jin, Jie and Ma, Ziyin and Yin, Min and Chen, Jinyu and Song, Haigang and Pang, Zhikun and Zhang, Xiaowen},
  year          = {2026},
  eprint        = {2610.02638},
  archivePrefix = {arXiv},
  note          = {Submitted to IEEE ICASSP 2027}
}

Built by AI降临派 / Adventists.ai. Claude (Anthropic) assisted with code.