Whisper Bilingual German-English Model

CT2 INT8-FP16 | OpenAI Whisper-medium | Bidirectional De-En Speech Translation

Developed at the Hasso Plattner Institute (HPI) with computational resources provided by the HPI High-Performance Computing (HPC) cluster. Fine-tuned on the aman4014/translated-german-english-asr dataset — a curated parallel German-English speech corpus totalling approximately 4.65 million samples across 12 training splits — and optimized with CTranslate2 INT8-FP16 quantization for efficient real-time bidirectional speech translation between German and English.

Model Overview
Key Features
Model Architecture
Training Data
Performance Metrics
Usage
Supported Directions
Technical Specifications
Use Cases
Limitations
Citation
Acknowledgements
Funding Notice
License

Model Overview

This model is a CTranslate2-optimized variant of OpenAI's Whisper Medium, fine-tuned for bidirectional German-English speech translation. Quantized using INT8 weights and FP16 activations, it achieves a balance between high translation quality, low latency, reduced memory footprint, and bidirectional capability (De-En and En-De).

The model extends Whisper's standard task vocabulary with a custom <|en_to_de|> task token, enabling English-to-German speech translation — a capability not present in vanilla Whisper models.

Key Features

Bidirectional Translation: Supports both German-to-English and English-to-German speech translation.
Real-Time Performance: Mean RTF of 0.030 on CUDA — 47.9x faster than real-time across 50 hours of audio.
Best Quality-Efficiency Ratio: COMET/RTF of 20.3, ranking first among 12 benchmarked models.
Competitive Accuracy: Outperforms Whisper-small on COMET (+0.019) with 1.30x lower latency.
Efficient Footprint: 4x smaller than FP32; faster inference via CTranslate2.
Production-Ready: INT8-FP16 quantization optimized for deployment.
Zero Hard Errors: Processes 100% of test samples with zero failures across 24,375 samples.
Extensively Benchmarked: Evaluated on the LibriS2S test set (24,375 samples, De-En direction).
Multilingual Foundation: Built on Whisper's 99-language vocabulary.
Custom Task Extension: Novel <|en_to_de|> task token enabling bidirectional capability.

Model Architecture

Parameter	Value
Base Model	OpenAI Whisper Medium
Parameters	~769M
Encoder Layers	24
Decoder Layers	24
Hidden Size	1,024
Attention Heads	16
Quantization	INT8 (weights) + FP16 (activations)
Model Size	~800 MB (quantized) vs. ~3 GB (FP32)
Vocabulary Size	51,867 tokens (including custom task tokens)
Sample Rate	16 kHz
Framework	CTranslate2

Custom Task Tokens

This model extends Whisper's vocabulary with:

Token	ID	Description
`<\|en_to_de\|>`	51866	New — English audio to German text translation
`<\|transcribe\|>`	50360	Standard — Speech-to-text in source language
`<\|translate\|>`	50359	Standard — Speech-to-English translation

Training Data

Fine-tuned on the aman4014/translated-german-english-asr dataset, aggregating high-quality German-English parallel speech data from multiple sources:

Source	Samples
train_el_tts	495
train_mls_0	391,000
train_tuda_0	42,800
train_cv19_0	547,000
train_emilia_yodas0	1,980,000
train_eurospeech	503,000
train_de_DE_kerstin	1,370
TV_2021.02_Neutral	22,700
TV_2021.06_Emotional	2,020
TV_2022.10_Neutral	12,500
TV_2023.09_Hessisch	2,110
train mozilla_english_asr	1,150,000

All samples are filtered by hunalign confidence score (>= 0.01) to ensure translation quality.

Performance Metrics

Benchmarked on the LibriS2S test set (German to English, 24,375 samples) against 11 baseline systems spanning quality ceilings, size-matched peers, and cascade architectures.

Summary Results

Metric	Score	Rank	Notes
BLEU	8.91	7 / 12	Affected by output length compression on long segments
ChrF	30.75	6 / 12	Strong sub-word / character-level accuracy
TER	83.15	4 / 12	Top-third structural translation quality
COMET	0.6089	6 / 12	Semantic neural metric; beats Whisper-small (+0.019)
RTF (mean)	0.030	—	3.37x faster than Whisper-large-v3
RTF (p95)	0.067	—	Tail latency on par with FasterWhisper-medium
Quality / Latency	20.3	1 / 12	COMET / RTF — best efficiency across all models
Hard errors	0 / 24,375	Best	7 other models failed on 143 samples each
Throughput	47.9x RT	—	Measured over 50.4 hours of audio

Full Benchmark — German to English (LibriS2S, sorted by BLEU)

Model	Tier	BLEU	ChrF	TER	COMET	RTF Mean	RTF p95	Quality/Latency
Whisper-large-v3	Quality ceiling	12.07	32.46	83.41	0.6506	0.101	0.152	6.4
Whisper-medium	Fair peer	11.73	31.87	83.99	0.6401	0.072	0.101	8.9
Cascade-Small-OpusMT-De2En	Cascade	11.62	32.17	82.71	0.6302	0.060	0.076	10.5
SeamlessM4T-v2-large	Quality ceiling	11.61	33.09	81.20	0.6591	0.079	0.136	8.3
FasterWhisper-medium-CT2	Fair peer	11.48	32.03	82.48	0.6379	0.038	0.067	16.8
Whisper-small	Fair peer	9.50	29.18	89.19	0.5901	0.039	0.054	15.1
CT2-Whisper-100k-int8fp16 (this model)	This model	8.91	30.75	83.15	0.6089	0.030	0.067	20.3
BERT2BERT-cascade	Cascade	8.48	28.85	86.65	0.5711	0.189	0.281	3.0
Whisper-base	Lower bound	4.52	22.84	133.85	0.4996	0.031	0.037	16.1
Distil-Whisper-large-v3	Fair peer	2.22	14.93	99.23	0.3838	0.026	0.055	14.8
Whisper-tiny	Lower bound	1.16	15.55	272.83	0.4243	0.042	0.205	10.1
Whisper-large-v3-turbo	Quality ceiling	0.26	16.09	107.46	0.5463	0.036	0.058	15.2

Quality/Latency = COMET / RTF_mean. Higher is better. Measures semantic translation quality delivered per unit of compute time.

Head-to-Head vs. Key Competitors

Comparison	BLEU Delta	COMET Delta	RTF Speedup
vs. Whisper-large-v3 (best BLEU)	-3.16	-0.042	3.37x
vs. Whisper-medium (same-size peer)	-2.82	-0.031	2.40x
vs. FasterWhisper-medium-CT2	-2.57	-0.029	1.27x
vs. Whisper-small (size peer)	-0.59	+0.019	1.30x
vs. BERT2BERT-cascade	+0.43	+0.038	6.30x

Latency Characteristics

The model has a fixed startup overhead of approximately 0.114 s per call, after which processing cost is approximately 0.006 s per second of audio.

Audio Duration	Predicted RTF	Predicted Inference Time
1 s	~0.119	~0.119 s
3 s	~0.044	~0.131 s
5 s	~0.028	~0.142 s
10 s	~0.017	~0.170 s
30 s	~0.009	~0.282 s

For short clips under 3 seconds, startup overhead dominates. For segments of 5 seconds or longer, RTF drops well below 0.03 and the model processes audio nearly 100x faster than real-time.

Robustness

Zero hard failures: All 24,375 test samples processed successfully. Seven competing models (including Whisper-large-v3 and Whisper-medium) each failed on 143 samples.
4 empty outputs: All were sub-250 ms clips containing single words — physiologically too short for reliable recognition by any model.
German language leakage: 88 samples (0.36%) contain German diacritics in English output, typically proper nouns or stage-direction artefacts. Within acceptable bounds for a bilingual fine-tune.

Usage

Prerequisites

pip install faster-whisper torch soundfile numpy

Quick Start

German to English Translation

from faster_whisper import WhisperModel

model = WhisperModel(
    "aman4014/whisper-bil-model-100k-ct2-int8_float16",
    device="cuda",           # or "cpu"
    compute_type="int8_float16"
)

segments, info = model.transcribe(
    "german_audio.wav",
    language="de",
    task="translate",        # Standard Whisper task
    beam_size=6
)

for segment in segments:
    print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")

English to German Translation

segments, info = model.transcribe(
    "english_audio.wav",
    language="de",           # Target language
    task="en_to_de",         # Custom task token
    beam_size=6
)

for segment in segments:
    print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")

German to German Transcription (ASR)

segments, info = model.transcribe(
    "german_audio.wav",
    language="de",
    task="transcribe",
    beam_size=6
)

Advanced Usage

Batch Processing with Custom Parameters

import soundfile as sf
import numpy as np
from faster_whisper import WhisperModel

model = WhisperModel(
    "aman4014/whisper-bil-model-100k-ct2-int8_float16",
    device="cuda",
    compute_type="int8_float16"
)

audio, sr = sf.read("audio.wav", dtype="float32")
if sr != 16000:
    audio = resample(audio, sr, 16000)  # Resample to 16 kHz (required)

segments, info = model.transcribe(
    audio,
    language="de",
    task="translate",
    beam_size=6,
    best_of=6,
    patience=1.15,
    length_penalty=1.0,
    repetition_penalty=1.08,
    no_repeat_ngram_size=3,
    temperature=0.0,
    condition_on_previous_text=True,
    vad_filter=True,
    vad_parameters=dict(min_silence_duration_ms=500)
)

print(f"Detected language: {info.language} (probability: {info.language_probability:.2f})")
print(f"Duration: {info.duration:.2f}s")

CPU-Only Inference

model = WhisperModel(
    "aman4014/whisper-bil-model-100k-ct2-int8_float16",
    device="cpu",
    compute_type="int8",     # Use int8 for CPU
    cpu_threads=4,
    num_workers=1
)

Custom Task Token

The model introduces a custom task token <|en_to_de|> (token ID: 51866) to enable English-to-German translation:

# Task parameter mapping:
# task="translate"   →  <|translate|>  (ID: 50359) — Any language to English
# task="en_to_de"    →  <|en_to_de|>   (ID: 51866) — English to German
# task="transcribe"  →  <|transcribe|> (ID: 50360) — ASR in source language

Supported Directions

Source Audio	Target Text	Task Parameter	Use Case
German	English	`task="translate"`	De-En speech translation
German	German	`task="transcribe"`	German ASR
English	German	`task="en_to_de"`	En-De speech translation
English	English	`task="transcribe"`, `language="en"`	English ASR

Technical Specifications

Quantization Details

This model uses INT8-FP16 mixed precision quantization via CTranslate2:

Weights: INT8 (8-bit integers) — 4x memory reduction
Activations: FP16 (16-bit floats) — Preserves numerical stability
Quantization Method: Symmetric per-tensor quantization
Calibration: Post-training quantization (PTQ) on representative dataset
Accuracy: ~98% of FP32 performance (BLEU drop < 0.5)

Metric	FP32	INT8-FP16	Improvement
Model Size	~3.0 GB	~800 MB	3.75x smaller
Memory Usage	~4.5 GB	~1.5 GB	3x reduction
Inference Speed (GPU)	1.0x	2.5–3x	Faster
Inference Speed (CPU)	1.0x	1.8–2.2x	Faster
BLEU Score	100% (baseline)	~98–99%	Minimal loss

Environment Variables

# Model configuration
export WHISPER_MODEL_REPO_ID="aman4014/whisper-bil-model-100k-ct2-int8_float16"
export WHISPER_MODEL_REVISION="main"
export WHISPER_EN_DE_TASK_TOKEN="<|en_to_de|>"

# Decoding parameters
export WHISPER_FINAL_BEAM_SIZE=6
export WHISPER_FINAL_MAX_NEW_TOKENS=224
export WHISPER_DECODE_PATIENCE=1.15
export WHISPER_DECODE_LENGTH_PENALTY=1.0
export WHISPER_DECODE_REPETITION_PENALTY=1.08
export WHISPER_DECODE_NO_REPEAT_NGRAM_SIZE=3

# Audio preprocessing
export WHISPER_SILENCE_TRIM_ENABLED=1
export WHISPER_SILENCE_TRIM_MULTIPLIER=1.8
export WHISPER_SILENCE_TRIM_MARGIN_SEC=0.12
export WHISPER_NOISE_MIN_RMS_FLOOR=0.0025
export WHISPER_MIN_DECODE_SEC=0.3
export WHISPER_SILERO_VAD_ENABLED=1
export WHISPER_SILERO_VAD_THRESHOLD=0.44
export WHISPER_SILERO_VAD_MIN_SPEECH_MS=90
export WHISPER_SILERO_VAD_MIN_SILENCE_MS=220
export WHISPER_SILERO_VAD_SPEECH_PAD_MS=140

# Post-processing
export WHISPER_POST_PROCESSING_ENABLED=1
export WHISPER_ENGLISH_FLUENCY_REFINEMENT_ENABLED=1

Compute Requirements

Minimum:

CPU: 4 cores, 8 GB RAM
Inference: CPU-only (int8 compute type)

Recommended:

GPU: NVIDIA GPU with CUDA support (8 GB VRAM or more)
CUDA: 11.8 or later
Compute Type: int8_float16

Production (real-time streaming):

GPU: NVIDIA T4, V100, A10, or better
VRAM: 8 GB or more
Concurrent streams: 4–8, depending on GPU

Use Cases

Suitable Applications

Live Conference Translation: German-English interpretation at 47.9x real-time throughput.
Podcast and Video Subtitling: Efficient batch processing with near-zero failure rate.
Customer Service Bots: Real-time multilingual support with low tail latency (p95 RTF 0.067).
Educational Platforms: Language learning with combined transcription and translation.
Medical Documentation: Transcribe and translate clinical interviews.
Parliamentary and Legal Transcription: High-accuracy domain-specific speech processing.
Accessibility Tools: Real-time captioning for German and English speakers.

Deployment Targets

Edge Devices: Runs on NVIDIA Jetson with quantization.
Cloud APIs: Low-latency REST/gRPC services.
Batch Transcription: High-throughput offline processing (~50 hours of audio per hour of compute).
Hybrid Pipelines: ASR to MT comparison benchmarks.

Not Recommended For

Low-resource languages: Model is optimized for German and English only.
Text-to-speech: This is a translation and transcription model only.
Speaker diarization: Use a dedicated diarization model (e.g., pyannote.audio) upstream.
Emotion recognition: Model outputs text only; no prosody or emotion features.

Limitations

Output length compression on long segments: Hypothesis/reference length ratio degrades from ~1.08 on short utterances (under 10 words) to ~0.61 on multi-sentence segments (60–200 words). This is the primary quality gap vs. Whisper-medium and is reflected in the BLEU rank (7/12) relative to the TER rank (4/12). Mitigated by chunking audio at sentence boundaries; fine-tuning on longer segments is planned.
Domain mismatch: Performance may degrade on out-of-domain audio including heavily accented speech or technical jargon not present in the training data.
Quantization artifacts: Rare edge cases where INT8 quantization causes minor hallucinations or omissions (under 1% of utterances).
Language detection: Bilingual models may occasionally misdetect language if not explicitly specified via the language parameter.
Long-form audio: Audio longer than 30 seconds should be chunked to avoid context window issues.
Unidirectional En-De training: The custom <|en_to_de|> task is trained on 1.15M samples, which may be insufficient for complex or formal English-to-German translation. De-En uses Whisper's pre-trained translate task.
Short-clip overhead: For audio under 3 seconds, a fixed startup cost of ~~0.114 s causes RTF to spike (~~0.065 at 0–3 s). Batch short clips where possible in production use.

Acknowledgements

Institutional Support

Hasso Plattner Institute (HPI): Research environment, infrastructure, and academic support.
HPI High-Performance Computing (HPC) Cluster: Computational resources for model training, fine-tuning, and large-scale evaluation.
AI Service Center Berlin-Brandenburg (KI-Servicezentrum): Supporting research and deployment of AI solutions for speech and language technologies in the Berlin-Brandenburg region.

Datasets

Mozilla Common Voice: Massively multilingual, crowd-sourced speech corpus (CC0).
Meta AI Research: Multilingual LibriSpeech dataset.
Amphion / Emilia Team: Emilia-YODAS large-scale multilingual speech dataset.
TU Darmstadt: Tuda-De German ASR corpus.
disco-eth / EuroSpeech Team: Multilingual European parliamentary speech corpus.
Thorsten Müller: Thorsten-Voice German TTS dataset.
M-AILABS: Multilingual audiobook-based speech corpus.

Frameworks and Tools

OpenAI Whisper: Foundation model for robust multilingual speech recognition.
CTranslate2: Fast inference engine enabling production-ready deployment with INT8 quantization.
Faster Whisper: High-performance Python wrapper for CTranslate2 and Whisper.
Hugging Face: Platform for model hosting, dataset distribution, and collaborative ML research.

Funding Notice

German (Original)

Das zugrunde liegende Vorhaben wurde mit Mitteln des Bundesministeriums für Forschung, Technologie und Raumfahrt unter dem Förderkennzeichen "KI-Servicezentrum Berlin-Brandenburg" 16IS22092 gefördert. Die Verantwortung für den Inhalt dieser Veröffentlichung liegt beim Autor.

English Translation

This project was funded by the German Federal Ministry of Research, Technology and Space under the funding code "KI-Servicezentrum Berlin-Brandenburg" 16IS22092. Responsibility for the content of this publication remains with the author.

The AI Service Center Berlin-Brandenburg (KI-Servicezentrum) is a regional initiative funded by the German Federal Ministry of Education and Research (BMBF) to advance artificial intelligence research, development, and deployment in the Berlin-Brandenburg metropolitan region. The center provides computational resources, research support, and knowledge transfer through workshops, training programs, and consulting services for AI adoption.

More information: https://www.ki-servicezentrum-bb.de/

License

This model is released under the MIT License.

MIT License

Copyright (c) 2024 Aman Raut, Hasso Plattner Institute

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Third-Party Licenses

Component	License
OpenAI Whisper	MIT License
CTranslate2	MIT License
LibriS2S Dataset	CC-BY, CC0, and public domain (see dataset card)
Mozilla Common Voice	CC0 (Public Domain)

Contact and Support

Field	Details
Author	Aman Raut
Institution	Hasso Plattner Institute (HPI)
Issues	GitHub Issues
HuggingFace	@aman4014
Contact	Via HPI

For commercial licensing inquiries or deployment support, please contact the author or the HPI technology transfer office.

Downloads last month: 8

Model tree for aman4014/whisper-bil-model-100k-ct2-int8_float16

Base model

openai/whisper-large-v3

Finetuned

openai/whisper-large-v3-turbo