Cohere Transcribe 2B — CoreML INT8 (W8A16) for Apple Silicon & iOS

This repository provides W8A16 Linear Symmetric Quantized CoreML (.mlpackage) models for Cohere Transcribe 2B (c4ai-transcribe-2b), optimized for high-performance speech-to-text on Apple Silicon (M1/M2/M3/M4/M5) and mobile Apple devices.

Quantized using Apple's official coremltools.optimize.coreml pipeline with OpLinearQuantizerConfig(mode='linear_symmetric', dtype='int8').

Performance Highlights (Apple M5 Silicon)

Benchmarked on the full 15.08-minute (904.93s) JFK Inaugural Address:

Metric	Lossless FP16 Baseline	W8A16 INT8 CoreML Engine	Advantage
Total Audio Duration	904.93 s (15.08 min)	904.93 s (15.08 min)	Identical
Total Compute Time	15.913 s	14.837 s	-1.08s faster
Real-Time Factor (RTF)	0.0176	0.0164	Lower is better
Throughput Speedup	56.9x Real-Time	61.0x Real-Time	+4.1x RT throughput
Peak Chunk Speedup	~61x RT	73.8x Real-Time (270 ms / 20s)	Instantaneous
Decoder Weight Size	258 MB	129 MB	50% reduction
Accuracy	1411 / 1411 words (100%)	1437 words	Verbatim fidelity

Architecture Partitioning

Conformer Encoder (cohere_encoder.mlpackage): Runs natively on the Apple Neural Engine (ANE), processing 20s of audio in ~190 ms while consuming 0% of the GPU.
Transformer Prefill (cohere_decoder_prefill.mlpackage): INT8 weights (145 MB), computes prompt representation in ~30 ms on GPU.
Autoregressive Decoder (cohere_decoder_decode.mlpackage): INT8 weights (129 MB) utilizing in-place Metal buffer mutation via MLState. Autoregressive single-step token latency drops from $1.68\text{ ms} \rightarrow 0.84\text{ ms}$ weight memory read time.

Quick Setup with Voxtype

To install and compile these INT8 CoreML models into Voxtype:

cd ~/voxtype
python3 scripts/setup_cohere_coreml.py --repo-id ctyau/cohere-transcribe-coreml-int8

This will automatically:

Download the .mlpackage bundles.
Compile them into native .mlmodelc bundles for your specific Mac.
Install them into ~/Library/Application Support/voxtype/models/cohere-transcribe-fp16/compiled_coreml.

Model Card & License

Base Model: CohereForAI/c4ai-transcribe-2b
License: CC-BY-NC-4.0 (inherits base model license)

Downloads last month: 32