RefWhisper

Model weights for RefWhisper, from the DLfM 2023 paper "Aligning Incomplete Lyrics of Korean Folk Song Dataset using Whisper".

RefWhisper is Whisper large-v2 with an extra reference encoder and a reference cross-attention layer in the decoder. It transcribes sung lyrics while consulting a pre-transcribed lyric that may be incomplete or partly wrong, and the reference cross-attention also aligns that lyric with the transcription and the audio.

File

File	Contents
ref_whisper_weight.pt	Training checkpoint with model (state dict), optim (optimizer state) and dims (model dimensions). fp32, 11.3 GB. The same file that was previously shared on OneDrive.

Model

• Initialized from whisper-large-v2. The audio encoder is frozen.
• Reference encoder: token embedding initialized from the Whisper decoder, a learnable positional encoding with a 256-token context, and two residual blocks of 1D convolutions (kernel size 3, 1280 channels).
• Decoder: the last 16 of the 32 blocks get a reference cross-attention layer and are fine-tuned. The earlier blocks and the token embedding are frozen.
• Training data: synthesized speech (Google Speech API) of two consecutive lines from the lyrics of the Anthology of Korean Traditional Folksongs. The reference input is the annotated lyric of the song, with its line order shuffled.
• AdamW, learning rate 1e-4 for new parameters and 3e-6 for pre-trained ones, batch size 8. This checkpoint was saved at the end of epoch 12 (70,776 steps).

Usage

Install the code from GitHub and run it from the repository root.

git clone https://github.com/MALerLab/RefWhisper.git && cd RefWhisper
pip install pipenv && pipenv install && pipenv shell
wget https://huggingface.co/MALer-Lab/refwhisper/resolve/main/ref_whisper_weight.pt

The reference lyric is a plain text file with one lyric line per line.

python -m ref_whisper.inference_timestamped song.wav song.txt --model ref_whisper_weight.pt --language ko --output_dir results

from ref_whisper.inference_timestamped import load_model, transcribe_timestamped

model = load_model("ref_whisper_weight.pt", device="cuda")

# transcription guided by the reference lyric
result = model.transcribe(audio="song.wav", reference_text="song.txt", temperature=0, language="ko")

# transcription with word timestamps and line-level alignment to the reference lyric
result = transcribe_timestamped(model, "song.wav", "song.txt", language="ko")
print(result["ref_align_segments"])

Results

Transcription accuracy on 201 manually transcribed segments (under 30 s) from folk songs that were not used for fine-tuning (Table 2 of the paper):

Model	WER	CER
Whisper large-v2	0.722	0.384
Whisper fine-tuned on the synthesized data	0.804	0.478
RefWhisper	0.319	0.169

A later re-evaluation of this checkpoint on 352 surviving test segments gave WER 0.331 / CER 0.162 (Whisper large-v2: 0.744 / 0.380).

The transcriptions and alignments for 13,801 songs of the anthology, produced with this model, are in transcription_result_csvs.tar.gz.

License

MIT, the same as the code. The audio and lyrics of the Anthology of Korean Traditional Folksongs are not included; see the official website.

Citation

@inproceedings{han2023aligning,
  title     = {Aligning Incomplete Lyrics of Korean Folk Song Dataset using Whisper},
  author    = {Han, Danbinaerin and Kim, Daewoong and Jeong, Dasaem},
  booktitle = {Proceedings of the 10th International Conference on Digital Libraries for Musicology (DLfM)},
  pages     = {7--11},
  year      = {2023},
  address   = {Milan, Italy},
  publisher = {ACM},
  doi       = {10.1145/3625135.3625154}
}