androiddrew/naptrap-phonmatchnet-en

PhonMatchNet (independently trained, English) for naptrap

A text-defined keyword spotter. You give it a short audio clip and a keyword typed as text, and it returns the probability that the keyword was spoken. Choosing a new wake word needs no new recordings or training.

This is the default model for naptrap. The architecture follows PhonMatchNet (Lee & Cho, Interspeech 2023).

Independent work. These weights were trained independently by Drew Bednar using the BSD-3-Clause PyTorch training code that NCSOFT published for PhonMatchNet, with modifications. They are not the paper authors' weights, and this release is not affiliated with or endorsed by NCSOFT or the PhonMatchNet authors. NCSOFT's license notice is in licenses/PhonMatchNet-BSD-3-Clause.txt.

Quick start

pip install "naptrap[hub]"

from naptrap import load_model, load_wav

model = load_model("phonmatchnet")   # downloads this repo and its two encoders
result = model.detect(load_wav("clip.wav"), "hey computer")
print(result.score, result.accepted)

Or from the command line: naptrap detect clip.wav "hey computer". Clips must be mono 16 kHz, 0.5–2 seconds.

How it works

Input	Source	Shape
Keyword text	g2p_en decoder hidden states (frozen)	[P, 256]
Audio, learned branch	40-bin log-mel, 10 ms hop (buffers stored here)	[T, 40]
Audio, pretrained branch	Google speech_embedding (frozen)	[T/8, 96]

A two-layer 1-D CNN encodes the mel features. It is added to the upsampled Google embeddings and concatenated with the projected text features. Causal single-head attention and a GRU then produce one keyword logit. The model has 324,993 parameters, not counting the two frozen encoders. The phoneme-level training head is removed for inference.

The repository contains:

• model.safetensors: the weights.
• config.json: thresholds, input contracts, and pinned checksums of the two encoders.
• examples/: a LibriSpeech clip of "said mister" and its features.
• conversion/export.py: how the files were produced from the training checkpoint.

Training

• Data: LibriSpeech train-clean-100 + train-clean-360 (CC BY 4.0), after Montreal Forced Aligner word alignment. That is 132,512 recordings from 1,172 speakers, split by speaker into 1,056 training, 58 development-selection and 58 calibration speakers. Development keywords and a strict held-out keyword list were excluded from training speech and text before fitting.
• Examples: 800,000 aligned one-to-four-word crops (200,000 per word count), crossed four ways with matching and non-matching keyword texts. That gives 3.2M trials per epoch.
• Objective: utterance-level BCE plus phoneme-position BCE.
• Optimizer: AdamW (lr 1e-4, weight decay 0.01), effective batch 2,048. At most 20 epochs, with patience 5.
• Selection: the checkpoint with the lowest keyword EER on the development-selection speakers (seed 17, step 10,941, epoch 7). Final test sets were not used for selection or calibration.
• Seeds: three seeds (17, 29, 43) were trained to measure repeatability. Seed 17 was fixed as the release model before final testing. Only seed 17 is published here.

Differences from the upstream code:

• a corrected crop frontend;
• explicit valid lengths, with BatchNorm computed over valid positions only;
• inference on one unpadded clip.

The numbers below describe these weights, not the results reported in the paper.

Evaluation

The task is content-only keyword detection (C-KWS): was the keyword spoken, by anyone? EER is the equal-error rate. FAR and FRR are per-clip false-accept and false-reject rates at the shipped far=0.01 threshold (score ≥ 0.7149).

Benchmark	Trials	EER, this model	EER, 3-seed mean ± SD	FAR / FRR at far=0.01
LibriPhrase Easy	270,684	1.38%	1.50 ± 0.15%	0.14% / 11.84%
LibriPhrase Hard	270,684	22.28%	22.81 ± 1.28%	31.00% / 12.76%
Speech Commands v1, 10 commands	236,608	13.29%	12.84 ± 0.85%	5.63% / 31.31%
Speech Commands v1, testing list	25,660	13.27%	12.60 ± 0.89%	5.35% / 31.88%

Reading these numbers:

• Many LibriPhrase keywords also appeared in the training text. On a strict subset with keywords held out from training (1,384 Easy / 1,098 Hard trials), the EER was 3.18% / 21.49%. Speech Commands is fully held out.
• The threshold does not transfer reliably. It was fitted for 1% FAR on development speakers. On LibriPhrase Hard's phonetically similar negatives, it accepts 31% of them. Calibrate your own threshold on audio from your deployment if false accepts matter. result.score is available for that.
• These are per-clip rates on pre-cut clips. They are not false activations per hour on continuous audio, which has not been measured.

Intended use and limitations

• Intended use: research, prototyping and hobby voice interfaces that need English user-chosen keywords on device.
• English only. The training speech is read audiobooks (LibriSpeech). Expect lower accuracy for conversational, far-field, noisy, accented or child speech.
• One clip at a time. Clips must be 0.5–2 s. naptrap does not yet provide streaming, voice activity detection or endpointing. For continuous audio, you need your own windowing.
• No speaker verification. Anyone saying the keyword triggers it.
• Phonetically similar phrases are often confused (see LibriPhrase Hard).
• Keywords are converted from spelling to sound by a neural G2P model. Unusual names may be pronounced differently from how users say them.
• Not suitable for security, authentication or safety-critical triggering.

License

The weights are released under the Apache License 2.0 (LICENSE, NOTICE). The training code they were produced with is © NCSOFT Corporation, BSD-3-Clause (licenses/PhonMatchNet-BSD-3-Clause.txt). The training data is LibriSpeech, CC BY 4.0. The two companion encoders are Apache-2.0 and are distributed in their own repositories.

Citation

If you use this model, please cite the PhonMatchNet paper:

@inproceedings{lee23phonmatchnet,
  title     = {PhonMatchNet: Phoneme-Guided Zero-Shot Keyword Spotting for User-Defined Keywords},
  author    = {Lee, Yong-Hyeok and Cho, Namhyun},
  booktitle = {Proc. INTERSPEECH 2023},
  pages     = {3964--3968},
  year      = {2023},
  eprint    = {2308.16511},
  archivePrefix = {arXiv}
}