Update README.md

README.md

@@ -1,7 +1,9 @@
-# `wangchanberta-base-wiki-spm`
+# WangchanBERTa base model: `wangchanberta-base-wiki-spm`
 
-Pretrained RoBERTa BASE model on Thai Wikipedia corpus.
+<br>
 
+Pretrained RoBERTa BASE model on Thai Wikipedia corpus.
+The script and documentation can be found at [this reposiryory](https://github.com/vistec-AI/thai2transformers).
 <br>
 
 ## Model description
@@ -62,6 +64,8 @@ You can use the pretrained model for masked language modeling (i.e. predicting a
 
 <br>
 
+The getting started notebook of WangchanBERTa model can be found at this [Colab notebook](https://colab.research.google.com/drive/15GdAgMcTxhWfDia0OB76F1l0fKZ059Ez?usp=sharing#scrollTo=eVlb6CEDmYi3)
+
 <br>
 
 ## Training data
@@ -84,12 +88,22 @@ Regarding the vocabulary, we use subword token trained with [SentencePice](https
 
 We sample sentences contigously to have the length of at most 512 tokens. For some sentences that overlap the boundary of 512 tokens, we split such sentence with an additional token as document separator. This is the same approach as proposed by [[Liu et al., 2019]](https://arxiv.org/abs/1907.11692) (called "FULL-SENTENCES"). 
 
-The details of the masking procedure for each sentence are the following:
+Regarding the masking procedure, for each sequence, we sampled 15% of the tokens and replace them with<mask>token.Out of the 15%, 80% is replaced with a<mask>token, 10% is left unchanged and 10% is replaced with a random token.
 
+<br>
 
 **Train/Val/Test splits**
 
 We split sequencially 944,782 sentences for training set, 24,863 sentences for validation set and 24,862 sentences for test set.
 
+<br>
+
+**Pretraining**
+
+The model was trained on 32 V100 GPUs for 31,250 steps with the batch size of 8,192 (16 mini batches per device with 16 accumulation steps) and a sequence length of 512 tokens. The optimizer we used is Adam with the learning rate of $7e-4$, $\beta_1 = 0.9$, $\beta_2= 0.98$ and $\epsilon = 1e-6$. The learning rate is warmed up for the first 1250 steps and linearly decayed to zero. The model checkpoint with minimum validation loss will be selected as the best model checkpoint. 
+
+<br>
+
+**BibTeX entry and citation info**
 
-<br>
+[Coming Soon]