alignment_heads is wrong in ctranslate2/config.json
First, thanks a lot for sharing this awesome work :)
alignment_heads in generation_config.jsonand ctranslate2/config.json are wrong : they involve more decoder layers than what there are really in the network.
This is causing a memory error when decoding with faster_whisper/ctranslate2 with alignement (i.e. option word_timestamps=True)
See https://github.com/SYSTRAN/faster-whisper/issues/688
This problems holds for all distilled models bofenghuang/whisper-large-v3-french-distil-dec*
Hi @Jeronymous ,
Thanks a lot for spotting this!
For now, I will leave them in the last half of the decoder layers, before conducting further tests to find the best alignment heads.
Excellent! Thanks
leave them in the last half of the decoder layers
That seems to be a first good approximation.
It what does ctranslate2 conversion by default when there is no specific alignment heads.