Steveeeeeeen/cohere-streaming

Cohere Streaming (2B)

A cache-aware streaming version of Cohere Transcribe (2B, offline). Words appear while the person is speaking. When they stop, a second pass produces the final transcript, close to offline accuracy.

• One model, five latencies: live-caption lookahead of 80, 160, 320, 560 or 1,120 ms, chosen at load time.
• Final transcript doesn't depend on the latency setting: at the end of speech the buffered utterance is re-encoded at 1.12 s lookahead and decoded with a 4-beam search, about 0.3 s after the speaker stops.
• 8 languages: English, French, German, Spanish, Italian, Portuguese, Dutch, Polish.

Results

All Open ASR Leaderboard numbers below use the leaderboard's own data loading, normaliser and scoring (huggingface/open_asr_leaderboard @ 67e8bd6). The other systems' numbers are the leaderboard's published ones. Our final transcript is the same at every latency setting, within ±0.05 average WER.

English (8 public datasets, WER %)

	AMI	Earnings22	GigaSpeech	LS clean	LS other	SPGISpeech	Monsoon	VoxPopuli	Avg
Cohere Streaming (2B)	9.80	14.38	10.30	2.03	4.72	3.91	7.76	5.19	7.26
Nemotron 3.5 ASR streaming (0.6B)	13.43	15.30	9.86	2.83	6.79	3.27	7.30	4.24	7.88
Voxtral Mini 4B Realtime	13.34	9.31	8.80	1.62	4.94	2.23	8.86	2.60	6.46
Cohere Transcribe, offline (not streaming)	7.02	7.88	7.88	0.97	2.05	2.74	3.44	5.38	4.67

Multilingual (average of FLEURS / Common Voice / MLS per language, WER %)

	fr	de	es	it	pt	nl	Avg
Cohere Streaming (2B)	6.91	6.41	4.63	5.48	10.19	9.78	7.23
Nemotron 3.5 ASR streaming (0.6B)	9.40	8.96	5.43	8.66	8.25	14.50	9.20
Voxtral Mini 4B Realtime	7.78	6.34	4.18	6.41	6.22	9.19	6.69
Cohere Transcribe, offline	4.02	3.10	2.81	2.97	5.99	4.17	3.84

Latency (LibriSpeech, real-time streaming harness, one A100, one stream)

Setting	WER clean / other	Median word delay (clean / other)	Final text after end of speech	Real-time factor, 1 stream
80 ms	1.36 / 4.85	303 / 356 ms	364 ms	0.68×
320 ms	1.36 / 4.85	263 / 296 ms	302 ms	1.69×
560 ms	1.36 / 4.85	387 / 429 ms	296 ms	2.46×
1120 ms	1.36 / 4.85	680 / 725 ms	321 ms	2.99×

Word delay is measured against forced-aligned word ends, and includes compute time. At 80 ms the per-stream Python loop is slower than real time on one A100. Use 320 ms or more, or a faster GPU, for live use.

Usage

The checkpoint includes its streaming model code, processor, and generation settings. Load it directly with Transformers:

pip install "transformers==5.14.1" torch librosa soundfile
hf auth login

from transformers import AutoModel

model = AutoModel.from_pretrained(
    "Steveeeeeeen/cohere-streaming",
    trust_remote_code=True,
    lookahead=6,
)
model.reset(language="en")
for chunk in audio_chunks_80ms:  # Mono float32 NumPy arrays: 16 kHz, 1,280 samples each.
    print(model.step(chunk))
print(model.finish())

The loader chooses CUDA, Apple Metal, or CPU. Pass device="cuda" or device="cpu" to choose explicitly, and dtype=torch.bfloat16 or dtype=torch.float32 to choose precision. Streaming supports one device. reset() starts an utterance, step() updates its partial transcript, and finish() flushes the remaining audio and decodes the endpoint.

Inference loads the weights and processor from this public repository without an authentication token; it does not download the original Cohere weights. Access to the original gated Cohere model is still needed for teacher labeling and distillation.

trust_remote_code=True loads the streaming implementation shipped here. This adapter exposes the streaming methods above; standard ASR pipeline streaming and AutoModel.from_config() are not implemented.

To save a copy with its code and processor:

model.save_pretrained("cohere-streaming-local")

The directory contains student.safetensors, config.json, mel_stats.json, model.py, processor/tokenizer files, and generation_config.json. The published repository also retains log.json from training. The local CohereStreaming adapter in the smol-audio example remains compatible with this checkpoint.

How it was made

1. Exact conversion of Cohere's 48-layer FastConformer encoder into transformers' cache-aware streaming encoder:
   • BatchNorm folded into a per-channel affine;
   • subsampling made causal in time;
   • parity checked to 1e-4 in non-causal mode.
2. Distillation from offline Cohere. About 12.5k hours were transcribed by the offline model, and the streaming student was trained on those transcripts. Losses: decoder cross-entropy, CTC, and cosine feature distillation at 4 encoder layers. Each batch samples one lookahead from {0, 1, 3, 6, 13}.
   • Read speech: LibriSpeech, MLS, VoxPopuli, Common Voice, FLEURS-train.
   • Conversational speech: about 1,460 h of AMI, GigaSpeech and People's Speech.
3. Robustness:
   • causal running feature normalisation with a randomised warm-up prior;
   • ±10 dB gain augmentation.
4. Endpoint decoding, tuned on dev sets:
   • re-encode the utterance at 1.12 s lookahead;
   • 4-beam search;
   • bracket tokens suppressed.

Limitations

• The final text takes about 0.3 s after end of speech. Nemotron takes about 50 ms.
• Live partials can be rewritten before they settle, especially at 80–320 ms on noisy audio.
• Transcription style follows Cohere's: lightly edited, not verbatim. This costs WER on verbatim references (VoxPopuli-cleaned) and on fully edited ones (SPGISpeech, where it keeps "you know").
• Weaker spots: financial vocabulary (Earnings22), Portuguese, and Common Voice Dutch.
• Long inputs: utterances over 30 s fall back to the streamed states for the final pass, and the final decode is capped at 256 tokens. Segment long audio with a VAD.

License and attribution

Apache-2.0, like the base model CohereLabs/cohere-transcribe-03-2026. The training audio comes from the datasets listed above, under their own licenses.