Prosody2text

Two signals — interpolated f0 and intensity at 200 Hz — into a mel spectrogram, and the mel into words. No phonetic content at any point: the model sees only how the pitch moved and how loud it was.

That is the claim this repo exists to let you check.

Companion to the paper. The interactive version, where you draw a contour instead of speaking one, is at Daporte/Prosody2text.

Use it

from transformers import AutoModel

model = AutoModel.from_pretrained(
    "Daporte/prosody2text-weights", trust_remote_code=True
)

result = model.transcribe("utterance.wav")
print(result.text)              # the chosen hypothesis
print(result.candidates)        # the whole beam -- ten of them
print(result.mel.shape)         # the predicted mel
print(result.features.f0_hz)    # what the conformer was actually given
print(result.fallback)          # True if this is the no-speech boilerplate

trust_remote_code=True is needed because the architecture is not one of Transformers' own: a conformer and a fine-tuned Whisper held alongside each other, each loaded from its own checkpoint. The code it runs is the modeling files in this repo.

Already have the contours? Hand them over directly — physical units, Hz and dB on the 200 Hz grid:

result = model(f0_hz=f0, intensity_db=intensity_db)

The normalisation to the model's own scale happens inside, so it cannot be applied twice — which produces a mel that looks plausible and is wrong.

AutoModel.from_pretrained(..., whisper=False) loads the conformer alone: a 45 MB download rather than 3.1 GB, and enough to look at the predicted mel.

Evaluate on a dataset

evaluate_dataset.py runs the pipeline over any HuggingFace speech dataset and scores it against the reference transcripts. A microphone tells you the pipeline runs; a dataset tells you whether it is right.

pip install transformers torch numpy scipy safetensors \
            amfm-decompy==1.0.12.2 praat-parselmouth \
            datasets jiwer soundfile

hf download Daporte/prosody2text-weights evaluate_dataset.py --local-dir .

python evaluate_dataset.py \
    --dataset espnet/yodas-granary \
    --data_files data/en001/asr_only/00000000.parquet \
    --split train --count 3000 --fp16 \
    --save ours_en001.jsonl

With uv instead of pip, in an empty directory:

uv init --bare                 # uv add needs a pyproject.toml to add to
uv add transformers torch numpy scipy safetensors \
       amfm-decompy==1.0.12.2 praat-parselmouth \
       datasets jiwer soundfile
uv run hf download Daporte/prosody2text-weights evaluate_dataset.py \
    --local-dir .
uv run python evaluate_dataset.py --dataset espnet/yodas-granary \
    --data_files data/en001/asr_only/00000000.parquet \
    --split train --count 3000 --fp16 --save ours_en001.jsonl

uv run python, not python: the bare command is the system interpreter, which does not have any of the above installed.

Streaming by default, so this does not download the whole split. Each line of the output is one utterance:

{"prediction": "...", "candidates": ["...", "..."], "reference": "...", "id": "0"}

Those field names match the published inference runs', so comparing against them is a join on id rather than a translation.

Anything with an audio column and a transcript works. The transcript column is found from text, sentence, transcription, transcript or normalized_text; pass --text_column otherwise. --help lists the rest.

On a GPU

Whisper at beam 10 is about 1 s per utterance on an RTX 3090 against ~20 s on CPU, so a few thousand utterances is the difference between an hour and a day. --fp16 halves the VRAM and is a further 1.8× faster; it transcribed every template identically to fp32.

If it reports whisper: cpu on a machine that has a GPU, the usual cause is that libcuda.so.1 sits somewhere the dynamic loader does not search — nvidia-smi works, but torch reports no driver at all. The model detects this and prints the fix. On most distributions none of this arises.

What the numbers mean

WER is reported over the corpus, not averaged per utterance: it is total errors over total reference words, so averaging would weight a three-word reference the same as a thirty-word one and give a different, wrong number. The normalisation — punctuation removed, case folded — matches the published runs, because case and punctuation alone would otherwise count as errors and the figure would measure typography rather than the model.

Each line also prints the substitution/deletion/insertion split, which is worth reading: on this pipeline a hypothesis is usually the right length with the wrong words, and that is a different failure from dropping or inventing half the utterance.

What is here


`conformer/`	45 MB	f0 + intensity → mel
`whisper/`	3.1 GB	Whisper-medium fine-tuned on prosody-predicted mels
`tokenizer/`	5 MB	the stock whisper-medium tokenizer, vendored
`config.json`		the mel contract, and the `auto_map`
`modeling_prosody.py`, `configuration_prosody.py`		the architecture
`features.py`, `speech.py`, `conformer.py`, `conformer_model.py`, `whisper_decoder.py`		the pipeline
`evaluate_dataset.py`		dataset evaluation

Only inference files. A training checkpoint also carries optimizer state — 6 GB of Adam moments for this model — which nothing here reads.

Notes

The feature extraction is not interchangeable with sensible-looking alternatives. The yaapt settings, the half-frame zero-padding, the six-frame trim and the two clamps all move the contour, and the conformer only ever saw one combination of them. speech.py carries that combination and is checked bit-identical against the reference implementation.

The mel contract lives in config.json. mel_bins and length_ratio must match what the conformer was trained with; a wrong length_ratio mis-times the mel against the audio without failing, which is the worst way for it to be wrong.

Recordings are trimmed; dataset clips usually need not be. A live microphone hands over the room before and after the words, and yaapt finds a pitch in noise, so an untrimmed recording puts several hundred milliseconds of nothing into the mel. The trim cuts anything more than 18 dB under the loudest frame. --no_trim, or trim=False, turns it off — which is the closer match to the published runs.

The conformer is the trained module, not an export. generate() is non-autoregressive and sizes its output from the real input length, so a phrase is predicted at its own duration. An earlier version of this repo shipped an ONNX export instead, whose input width is baked in at 2000 frames — every phrase padded to ten seconds, with the decoder attending over the padding. Measured against the training pipeline's own predictions that is 0.9357 correlation against 0.9985 here, which is two orders of magnitude more than the f0 extraction differs by. The architecture in conformer_model.py is extracted verbatim from the training code and checked bit-identical to it on CPU.

This is not trainable as it stands. It subclasses PreTrainedModel so that from_pretrained, the AutoModel API and download metrics work, but forward runs the two models in sequence under no_grad rather than building one differentiable graph, and the weights load from two checkpoints rather than one state dict. It exists to run the published pipeline.

Licence

MIT for the code. The weights derive from openai/whisper-medium and carry its terms.

Downloads last month: 126

Daporte
/

prosody2text-weights