NagaYu/vouch-hubert-lr

Vouch HuBERT-LR: a research detector for synthetic speech in short voice notes

A logistic-regression head on frozen facebook/hubert-base-ls960 features (revision dba3bb02fda4248b6e082697eee756de8fe8aa8a, layer 3). It returns a bona fide likelihood in [0, 1] (1 = real speech) for a 2-10 s voice note.

Read this first: limitations

• Do not use this score as the sole basis for any fraud decision or any decision about a person. It is a research artefact with known blind spots.
• It cannot be trusted on a voice-conversion family it was not trained on. This checkpoint saw two VC systems (kNN-VC, FreeVC). When the method is trained with no VC at all and tested on VC, AUC is 0.503 on the research benchmark (chance) and 0.351 in this checkpoint's data setup (below chance, so decisions invert). Even within the VC lineage, an unseen system is not reliably caught: with FreeVC in training and kNN-VC unseen, AUC is 0.515 in this checkpoint's data setup (chance) and 0.635 on the research benchmark.
• Zero-shot voice cloning has not been measured. Neither have commercial TTS services.
• Scores are not calibrated. At a threshold of 0.5, 48% of clips from a modern neural TTS never seen in training (Kokoro-82M) pass as real with this checkpoint. The figure is 44% averaged over the 3 release seeds and 39% for the research-benchmark models.
• Japanese numbers are exploratory. There are few speaker groups, and the Japanese test speakers are female only.
• Phone-microphone audio is the weakest condition, with the highest EER in every per-condition breakdown below.

Evaluation

All values are EER / AUC on the test side of each fold. Real speech is the positive class. Values are the mean ± sd over 3 seeds; baselines are a single deterministic run. Numbers are copied mechanically from the files in eval/, and the last column names the exact key.

(a) Benchmark results: vouch-bench v2, research setting

These models were trained with macOS-say clips, which are not redistributable. They are not this checkpoint; the numbers describe the method.

model	All seen	espeak-ng unseen	macOS say unseen	kNN-VC unseen	FreeVC unseen	VC lineage unseen	TTS lineage unseen	source
Duration-only control	0.492 / 0.488	0.508 / 0.460	0.420 / 0.508	0.586 / 0.398	0.586 / 0.402	0.571 / 0.408	0.460 / 0.508	eval/benchmark_v2_summary.json folds["duration/"]
MFCC + LR baseline	0.288 / 0.801	0.836 / 0.135	0.265 / 0.770	0.571 / 0.380	0.452 / 0.540	0.591 / 0.324	0.539 / 0.438	eval/benchmark_v2_summary.json folds["mfcc/"]
HuBERT + LR	0.069 ± 0.009 / 0.985 ± 0.003	0.048 ± 0.021 / 0.993 ± 0.007	0.131 ± 0.023 / 0.953 ± 0.018	0.407 ± 0.046 / 0.635 ± 0.051	0.202 ± 0.010 / 0.860 ± 0.007	0.494 ± 0.003 / 0.503 ± 0.001	0.279 ± 0.006 / 0.803 ± 0.017	eval/benchmark_v2_summary.json folds["ssl_lr/"]
HuBERT + MLP	0.085 ± 0.012 / 0.976 ± 0.003	0.061 ± 0.028 / 0.985 ± 0.013	0.122 ± 0.012 / 0.948 ± 0.003	0.444 ± 0.048 / 0.589 ± 0.053	0.363 ± 0.096 / 0.709 ± 0.098	0.493 ± 0.009 / 0.524 ± 0.017	0.341 ± 0.090 / 0.732 ± 0.105	eval/benchmark_v2_summary.json folds["ssl_mlp/"]

(b) This checkpoint: release evaluation

Trained and evaluated only on the public subset (no macOS say). The "say unseen" fold cannot be evaluated in this setting. family-tts equals logo-espeak_ng, because espeak-ng is the only trainable TTS.

model	All generators seen	espeak-ng unseen (= TTS lineage unseen)	kNN-VC unseen (FreeVC in train)	FreeVC unseen (kNN-VC in train)	VC lineage unseen (both VCs held out)	TTS lineage unseen	source
Duration-only control	0.537 / 0.461	0.478 / 0.519	0.529 / 0.462	0.526 / 0.467	0.557 / 0.440	0.478 / 0.519	eval/release_summary.json folds["duration/"]
MFCC + LR baseline	0.277 / 0.807	0.897 / 0.045	0.556 / 0.440	0.500 / 0.502	0.591 / 0.328	0.897 / 0.045	eval/release_summary.json folds["mfcc/"]
HuBERT + LR	0.052 ± 0.009 / 0.989 ± 0.001	0.256 ± 0.016 / 0.835 ± 0.017	0.488 ± 0.009 / 0.515 ± 0.019	0.189 ± 0.004 / 0.887 ± 0.009	0.594 ± 0.010 / 0.351 ± 0.007	0.256 ± 0.016 / 0.835 ± 0.017	eval/release_summary.json folds["ssl_lr/"]

The released file is the seed-0 model of the speaker fold (test EER 0.046). Per-condition EER of the HuBERT + LR row on the speaker fold (eval/release_summary.json per_condition["ssl_lr"]): clean 0.033 ± 0.008, Opus 16k 0.033 ± 0.013, Opus 24k 0.040 ± 0.008, phone mic 0.098 ± 0.013; English 0.056 ± 0.006, Japanese 0.057 ± 0.007 (exploratory).

Modern-TTS hold-out (Kokoro-82M, never used for training or selection)

speaker-fold models, scored on that fold's test-side real clips plus all Kokoro clips.

model	EER / AUC	length-matched	EER en	EER ja (expl.)	EER phone mic	Kokoro passing at 0.5	real passing at 0.5	source
Benchmark HuBERT + LR (with say)	0.237 ± 0.030 / 0.850 ± 0.032	0.204 ± 0.040 / 0.871 ± 0.034	0.279 ± 0.048	0.103 ± 0.019	0.366 ± 0.054	39%	90%	eval/benchmark_v2_summary.json holdout["ssl_lr/speaker"]
Release setup, HuBERT + LR (3 seeds)	0.225 ± 0.002 / 0.870 ± 0.005	0.194 ± 0.011 / 0.888 ± 0.004	0.251 ± 0.032	0.105 ± 0.017	0.341 ± 0.012	44%	92%	eval/release_summary.json holdout["ssl_lr/speaker"]
This checkpoint (seed 0)	0.226 / 0.871	0.202 / 0.885	0.233	0.125	0.333	48%	92%	eval/release_summary.json holdout["ssl_lr/speaker"] values[0]
MFCC + LR baseline	0.422 / 0.533	0.441 / 0.501	0.388	0.500	0.565	51%	65%	eval/release_summary.json holdout["mfcc/speaker"]
Duration-only control	0.429 / 0.546	0.540 / 0.469	0.414	0.441	0.429	48%	62%	eval/release_summary.json holdout["duration/speaker"]

Training data and protocol

• Data: the public subset of vouch-bench v0.2 (NagaYu/vouch-bench). Real speech: LibriSpeech dev-clean (English) and FLEURS ja_jp (Japanese). Fakes: espeak-ng, kNN-VC and FreeVC, each rendered clean, Opus 16k, Opus 24k and phone mic. This checkpoint trains on the train side of the speaker-disjoint speaker fold.
• Leave-one-generator-out (LOGO): each logo-* fold removes one generator from training entirely. Each family-* fold removes a whole lineage (all VC, or all TTS). Speakers never cross between train and test.
• Selection: a speaker-disjoint dev split is carved from the train side. Layer and C are chosen by dev EER only. The released seed was fixed by a rule written down before training: lowest dev EER among 3 seeds, ties to the lowest seed. All three seeds tied at dev EER 0.0391, so seed 0 was released.
• Inputs: 16 kHz mono, edge-silence trim, at most 10 s, -23 dBFS loudness, then three evenly spaced 3 s crops (repeat-padded when shorter). The score is the mean crop probability.
• History, honestly:
  • v1 (plain dev selection) inverted on unseen VC.
  • v1.1 (nested leave-one-generator-out selection) was designed after those results were seen. It helped on some folds and hurt on others, so it was not adopted.
  • v2 added a second VC lineage. That made within-lineage unseen VC measurable and partly recoverable. It did not help when no VC at all is in training, and adding VC data made the macOS-say-unseen fold worse.

Efficiency

• Parameters: 1,537 trainable (the head, 18.7 kB) plus a frozen 94.4M-parameter HuBERT (about 378 MB, downloaded once from the Hub at the pinned revision).
• Inference: about 275 ms per clip on CPU (Apple M2, median of 20 clips, preprocessing included).

How to use

pip install -r requirements.txt          # also needs ffmpeg on PATH
python predict.py --audio note.ogg

from vouch_detector import VouchDetector
det = VouchDetector(".")                  # downloads facebook/hubert-base-ls960 at the pinned revision once
det.score_file("note.ogg")                # {"score": ..., "crop_scores": [...], "duration_sec": ...}

Licence and attribution

The code and head weights are offered under MIT. The frozen front end, facebook/hubert-base-ls960, is Apache-2.0 and is not redistributed here. The training data is CC BY 4.0 (LibriSpeech, FLEURS; VCTK through FreeVC's training). Full component table: LICENSES.md in NagaYu/vouch-bench.

Contact

Portfolio: https://nagayu-portfolio.nagayu.workers.dev/en