Tabahi/CUPE-3-mls7

CUPE v3 (p3cupe) · multilingual (de, es, fr, it, nl, pl, pt)

A context-free CNN phoneme encoder (the third CUPE), trained without boundary labels to place phoneme boundaries: one ph66 phoneme distribution every 5 ms, each frame seeing at most 120 ms of audio (38.9 ms receptive field). Boundaries come from a segmental Viterbi decoder over the known phone sequence, read at the crossing of neighbouring posteriors.

Superseded by p4mbfa (Tabahi/mbfa-<language group>, standard_g2p labels, many languages). Kept for BFA's ph66 pipeline.

head	labels	size
phonemes	ph66 (config.json labels.phonemes, index 0 = SIL; the last slot, 'noise', is never trained)	67
groups	ph66 phoneme groups (labels.phoneme_groups)	17

Training

• Data: Multilingual LibriSpeech, MLS_7l_100h: 100 h each of German, Spanish, French, Italian, Dutch, Polish and Portuguese (700 h, no English), one epoch, training noise_level 0.01.
• Checkpoint: experiment pa07a, epoch 0, selected on TIMIT lev f1@25ms on the tune half, re-scored with the final decoder (best of the run's three saved checkpoints).

The only multilingual CUPE v3: the English CUPE-3-en-base (pa01d epoch 3) trained one more epoch on seven non-English MLS languages, with soft targets (soft_target_mix 1.0), min_phone_frames 6, soft 4, onset bonus 30, ramp 3. There are no boundary labels outside English, so the scores below are English TIMIT: they show how much English boundary accuracy survived (76.00 -> 74.15 holdout), not how well the seven languages align. Later epochs of this run kept falling on TIMIT (73.74 at epoch 1, 71.70 at epoch 4), so they are not published.

metric	value
timit_lev_f1_25ms_tune	74.79
timit_lev_f1_25ms_holdout	74.15
timit_lev_f1_25ms_bonus0_holdout	70.89
timit_phone_hit_25ms_bonus0_holdout	71.11
timit_phone_hit_25ms_holdout	66.41

TIMIT: 400 clips (English, never trained on), split in a tune half (picked the checkpoint and the decoder) and a holdout half. lev_f1_25ms is phone-boundary F1 within 25 ms; phone_hit_25ms is the share of phones whose onset is within 25 ms. Scored with the decoder in config.json ("decoder": min 6 + soft 10 frames, log-mel onset bonus 30.0); bonus0 = the same decoder without the bonus. The settings it was trained with are under training.decoder_at_training.

Usage

The code is in https://github.com/tabahi/bfa_models (p3cupe/); clone it and run from its root.

import torch
from p3cupe.model import CupeExtractor
from p3cupe.windowing import slice_windows, stitch_windows

ext = CupeExtractor("Tabahi/CUPE-3-mls7", device="cpu")         # or a release dir / checkpoint
wav = torch.randn(1, 1, 3 * 16000)                     # [B, 1, samples] at 16 kHz
wav = wav / wav.pow(2).mean().sqrt()                   # unit RMS per clip, as in training
win = slice_windows(wav, 16000, 120, 80)  # [B, n_windows, 1920]
b, n, _ = win.shape
logits_ph, logits_grp = ext.predict(win.reshape(b * n, -1))
stride_frames = 1280 // ext.frame_stride_samples
ph = stitch_windows(logits_ph.reshape(b, n, *logits_ph.shape[1:]), n,
                    ext.frames_per_window, stride_frames)  # [B, frames, 67], 5 ms frames

Forced alignment over a known ph66 sequence is p3cupe.loss.PeakLoss.align with the config.json decoder settings, as p3cupe/timit_probe.py does.

Training checkpoint

ckpt/mls7_mls700h_pa07a_e0_timit_f1_25ms=74.15.ckpt is the PyTorch Lightning checkpoint (weights only; a pickle, so load it only if you trust this repo), for ckpt_path: in p3cupe/cupe_hp.yaml or trunk_init_path: in p4mbfa.

License: AGPL-3.0.