ctyau/cohere-transcribe-coreml-fp16

Cohere Transcribe 2B — Native CoreML FP16 (Apple Neural Engine + MLState)

Native Apple Silicon CoreML FP16 release of CohereLabs/cohere-transcribe-03-2026 (2 Billion parameter Conformer + Transformer ASR model).

This release utilizes Apple's CoreML.framework with MLState in-place KV caching (iOS 18 / macOS 15+), achieving 26× to 31× realtime transcription speed and ~210 tokens/sec decoding throughput on Apple Silicon M5.

---

Performance Benchmark (Apple M5, 10-core, 32 GB RAM)

Utterance Duration	Processing Time	Realtime Factor (RTF)	Realtime Speedup	Decoder Step Latency	Decode Throughput
8.40s	0.320s	0.038	26.3× realtime	4.76 ms / token	~210 tok/sec
9.80s	0.310s	0.032	31.6× realtime	4.76 ms / token	~210 tok/sec
11.70s	0.430s	0.037	27.2× realtime	4.76 ms / token	~210 tok/sec

---

Architectural Highlights

1. In-Place MLState Key-Value Caching

Standard ONNX / PyTorch models require passing the full past key-value tensors in and out of the graph at each autoregressive step, incurring significant memory bandwidth and memory copy overhead.

In this CoreML release, the decoder is compiled as a stateful model. It allocates 32 on-chip state buffers:

• 16 cross-attention buffers (cross_k_0..7, cross_v_0..7)
• 16 self-attention buffers (self_k_0..7, self_v_0..7)

At decode time, the MLState object is mutated directly in Neural Engine and GPU SRAM without streaming tensors across memory domains. Single-token decode latency is reduced to 4.76 ms.

2. Length-Aware Conformer Encoder

The encoder accepts mel spectrogram [1, 128, 3500] and dynamic length [1]. By tracking the real acoustic duration, padding tokens are properly masked out, preventing repetition and hallucination loops common in naive static exports.

3. Complete Production Assets

Includes pre-computed Slaney 128-bin mel filterbank weights (mel_filters.bin) and complete tokenizer files (tokenizer.json, tokenizer_config.json, processor_config.json, config.json), making it a drop-in asset for voice applications.

---

Model Bundle Contents

File	Size	Description
cohere_encoder.mlpackage	~3.5 GB	Conformer ASR Encoder (FP16 weights, ANE accelerated)
cohere_decoder_prefill.mlpackage	~289 MB	Prompt prefill & cross-attention (initializes MLState)
cohere_decoder_decode.mlpackage	~258 MB	1-token autoregressive step (mutates MLState in place)
mel_filters.bin	102.9 KB	128-bin Slaney mel filterbank (16kHz, n_fft=400, IEEE float32)
cohere_transcribe_trace_metadata.json	1.6 KB	Tensor input/output metadata and shape specifications
tokenizer.json / config.json	~1.2 MB	Tokenizer vocab (16,384 tokens) and model hyper-parameters

---

Model Inputs & Outputs

1. Encoder (cohere_encoder.mlpackage)

• Inputs:
  • mel: [1, 128, 3500] (float32, log-mel spectrogram)
  • length: [1] (int32, actual number of acoustic frames)
• Output:
  • encoder_hidden: [1, 438, 1024] (float16 acoustic representations)

2. Decoder Prefill (cohere_decoder_prefill.mlpackage)

• Inputs:
  • encoder_hidden: [1, 438, 1024] (float32)
  • input_ids: [1, 10] (int32, prompt prefix: [13764, 7, 4, 16, 62, 62, 5, 9, 11, 13])
• State Output: Initializes MLState buffers
• Logits Output: [1, 10, 16384] (predicts initial token)

3. Decoder Step (cohere_decoder_decode.mlpackage)

• Inputs:
  • input_ids: [1, 1] (int32, current token)
  • cache_update_mask: [1, 512] (float32, one-hot update slot)
  • cache_valid_mask: [1, 512] (float32, cumulative mask)
• State Mutation: Updates MLState in place
• Logits Output: [1, 1, 16384]

---

Quick Compilation & Setup

Using Python coremltools:

from coremltools.models.utils import compile_model
from huggingface_hub import snapshot_download
import shutil, os

# Download
path = snapshot_download(repo_id="ctyau/cohere-transcribe-coreml-fp16")

# Compile to native .mlmodelc bundles
dest = os.path.expanduser("~/Library/Application Support/voxtype/models/cohere-transcribe-fp16/compiled_coreml")
os.makedirs(dest, exist_ok=True)

for name in ["cohere_encoder", "cohere_decoder_prefill", "cohere_decoder_decode"]:
    compiled_temp = compile_model(f"{path}/{name}.mlpackage")
    target = os.path.join(dest, f"{name}.mlmodelc")
    if os.path.exists(target):
        shutil.rmtree(target)
    shutil.copytree(compiled_temp, target)
    print(f"Compiled {name} -> {target}")

---

Voxtype Integration

This model bundle powers native Apple Silicon dictation in Voxtype:

• Zero-copy CoreAudio streaming.
• Sub-5ms autoregressive decoding.
• Global push-to-talk hotkey and floating waveform HUD.

---

License

Released under the Apache-2.0 license, consistent with the upstream base model CohereLabs/cohere-transcribe-03-2026.