BobJames/open-higgs-tokenizer

Higgs Audio V2 Tokenizer

Hybrid semantic + acoustic speech tokenizer. Encodes audio to discrete codes and decodes discrete codes to a 24 kHz waveform.

Model Details

Property	Value
Model type	higgs_audio_v2_tokenizer
Architecture	HiggsAudioV2TokenizerModel
Developed by	Higgs audio team
License	Apache-2.0
Language	English

Architecture

Component	Description
Semantic branch	7-layer strided convolutional feature extractor followed by a 12-layer transformer encoder (hidden size 768, 12 attention heads, intermediate size 3072); 16 kHz input, downsample factor 320, 50 frames/s
Acoustic branch	DAC-style convolutional encoder and decoder (hidden size 256, decoder hidden size 1024); downsampling ratios 8, 5, 4, 2, 3; hop length 960 at 24 kHz, 25 frames/s
Quantiser	Residual vector quantiser, 8 codebooks, codebook size 1024, codebook dimension 64, no quantiser dropout
Fusion	Linear projections fc, fc1, fc2 combine semantic and acoustic features before quantisation
Target bitrates	0.5, 1, 1.5, 2 kbps

Uses

Direct Use

• Encoding speech into discrete codes for analysis, compression, or speech language model input.
• Decoding discrete codes to 24 kHz audio.

Out-of-Scope Use

• Speaker identification.
• Voice cloning without consent.
• Music or general sound effects.

Bias, Risks, and Limitations

• English-focused model.
• Reconstruction quality varies with accent and channel conditions.

Recommendations

• Measure reconstruction on target accents and channel conditions before using the codes.
• Evaluate decoded audio with a speech recognizer when intelligibility matters.

Model Files and Format

Property	Value
File	model.safetensors
Format	safetensors, single file, no sharding
File size	805,665,628 bytes (805.7 MB, 768.3 MiB)
Tensors	527
Tensor dtype	float32
Parameters	201,400,553
Header size	63,408 bytes
Metadata	{"format": "pt"}
SHA-256	51c9b982f4e0d26907eb419ce2346a97b24583f7d30057c585f0847a894b8b6b

Parameter Split

Module	Parameters
semantic_model.*	94,370,944
acoustic_encoder.*	51,314,496
acoustic_decoder.*	20,238,113
encoder_semantic.*	14,747,136
decoder_semantic.*	16,516,608
quantizer.*	2,114,056
fc.*	1,049,600
fc1.*	787,200
fc2.*	262,400

Companion Files

File	Description
config.json	Model configuration
preprocessor_config.json	Feature extractor configuration
LICENSE	Apache-2.0 license text
WEIGHTS.md	Weight file size and checksum

Training Details

Training Data

Property	Value
Total audio	14,300 hours
Language	English
License	openly licensed
Sources	read audiobooks, broadcast news, conversational podcasts, telephone-style recordings
Telephone-style audio	~20% of total
Clip length	2 to 15 seconds
Filtering	overlapping speakers, clipping

Training Procedure

Hyperparameter	Value
Weight precision	float32
Mixed precision	bfloat16
Optimiser	AdamW
Peak learning rate	2e-4
Learning rate schedule	cosine decay
Warm-up steps	4,000
Batch size	64 clips (~6.4 minutes of audio)
Training steps	310,000
Reconstruction loss	reconstruction + multi-scale spectral loss
Codebook loss weight	1.0
Commitment loss weight	0.25
Semantic distillation loss weight	0.5
Best validation step	287,000

Checkpoint Size and Throughput

Property	Value
Checkpoint size	805.7 MB
Training throughput	~38 audio-hours per wall-clock hour
Selected checkpoint	best validation step, 287,000

Evaluation

Testing Data

Property	Value
Utterances	1,850
Audio	~4.1 hours
Speakers	46, disjoint from training
Clean studio clips	925
Telephone-style clips	925, simulated narrowband channel effects

Metrics

Metric	Definition	Direction
Mel distance	L1 distance between log-mel spectrograms	lower is better
WER	word error rate of a fixed reference recognizer on decoded audio	lower is better
Speaker similarity	cosine similarity of speaker embeddings	higher is better

Results

Condition	Mel distance	WER (decoded)	Speaker similarity
Clean studio	0.42	4.8%	0.91
Telephone-style	0.67	11.3%	0.84

Reference Deltas

Condition	WER increase vs. uncoded audio
Clean studio	1.1 percentage points
Telephone-style	3.9 percentage points

Codebook Usage

Property	Value
Codebook entries used	~93% per codebook on the test set
First two codebooks	most speech content
Later codebooks	timbre and channel noise refinement

Environmental Impact

Property	Value
Hardware	4 x 80 GB accelerators
Hours used	212
Cloud provider	regional cloud provider (unnamed)
Compute region	European region
Carbon emitted	~46 kg CO2eq

Carbon estimates can be reproduced with the Machine Learning Impact calculator from Lacoste et al. (2019).

Technical Specifications

Property	Value
Model type	higgs_audio_v2_tokenizer
Architecture	HiggsAudioV2TokenizerModel
Acoustic model type	dac
Semantic model type	hubert
Semantic feature extractor layers	7
Semantic transformer layers	12
Codebook loss weight	1.0
Commitment loss weight	0.25
Codebooks	8
Codebook size	1024
Codebook dimension	64
Quantiser dropout	0
Sample rate	24000
Semantic sample rate	16000
Serialisation	safetensors
Framework	Transformers 5.3.0.dev0, PyTorch

Model Card Contact

Open an issue on the model repository.