Create README.md

README.md

@@ -0,0 +1,19 @@
+---
+language:
+- sv
+
+---
+
+# Megatron-BERT-base Swedish 600k
+
+This BERT model was trained using the Megatron-LM library.
+The size of the model is a regular BERT-base with 110M parameters.
+The model was trained on about 70GB of data, consisting mostly of OSCAR and Swedish newspaper text curated by the National Library of Sweden.
+
+Training was done for 600k training steps. Its [sister model](https://huggingface.co/KBLab/megatron-bert-base-swedish-cased-125k) used the same setup, but was instead trained for only 125k steps.
+
+
+The model has three sister models trained on the same dataset:
+- [Megatron-BERT-base-125k](https://huggingface.co/KBLab/megatron-bert-base-swedish-cased-125k)
+- [Megatron-BERT-base-600k](https://huggingface.co/KBLab/megatron-bert-base-swedish-cased-600k)
+- [Megatron-BERT-large-110k]()