README.md

metadata

datasets:
  - oscar
  - srwac
  - leipzig
  - cc100
  - hrwac
language:
  - hr
  - sr
  - multilingual
tags:
  - masked-lm
widget:
  - text: Ovo je početak <mask>.
license: apache-2.0

Transformer language model for Croatian and Serbian

Trained on 43GB datasets that contain Croatian and Serbian language for one epochs (9.6 mil. steps, 3 epochs). Leipzig Corpus, OSCAR, srWac, hrWac, cc100-hr and cc100-sr datasets

Validation number of exampels run for perplexity:1620487 sentences Perplexity:6.02 Start loss: 8.6 Final loss: 2.0 Thoughts: Model could be trained more, the training did not stagnate.

Model	#params	Arch.	Training data
Andrija/SRoBERTa-X	80M	Fifth	Leipzig Corpus, OSCAR, srWac, hrWac, cc100-hr and cc100-sr (43 GB of text)