Support for new language.
Hi team,
can i add new language token and fine tune it?
i want to test how well the model learns new languages with sufficient data to train.
Hi,
Please inspect the tokenizer's vocab first. (The tokenizer can be extracted from the .nemo file available here: https://drive.google.com/drive/folders/1QdMG0R3oGgOThm7uqYeMnt8zGOSx467x)
If the characters of your target language's script is well represented in the vocab, you can finetune without changing the tokenizer or the decoder. A recipe is available here: https://huggingface.co/blog/ARTPARK-IISc/fine-tune-sravaani-on-your-own-speech-data
On the other hand, if language's script is missing from the vocab or is underrepresented, then you have to create a new tokenizer with your vocab and train a new decoder head (since the decoder's last layer is the vocab size, so whenever you change the tokenizer/vocab you have to change the decoder and train it again). In this case, only the encoder's learning will be carried forward, and it should be useful if your language is related to the languages SraVaani1.0 was trained with.