Transformer language model for Croatian and Serbian

Trained on 43GB datasets that contain Croatian and Serbian language for one epochs (9.6 mil. steps, 3 epochs). Leipzig Corpus, OSCAR, srWac, hrWac, cc100-hr and cc100-sr datasets

Validation number of exampels run for perplexity:1620487 sentences Perplexity:6.02 Start loss: 8.6 Final loss: 2.0 Thoughts: Model could be trained more, the training did not stagnate.

Model #params Arch. Training data
Andrija/SRoBERTa-X 80M Fifth Leipzig Corpus, OSCAR, srWac, hrWac, cc100-hr and cc100-sr (43 GB of text)
New

Select AutoNLP in the “Train” menu to fine-tune this model automatically.

Downloads last month
3
Hosted inference API
Fill-Mask
Mask token: <mask>
Examples
Examples
This model can be loaded on the Inference API on-demand.