|
--- |
|
inference: false |
|
language: id |
|
--- |
|
|
|
# IndoConvBERT Base Model |
|
|
|
IndoConvBERT is a ConvBERT model pretrained on Indo4B. |
|
|
|
## Pretraining details |
|
|
|
We follow a different training procedure: instead of using a two-phase approach, that pre-trains the model for 90% with 128 sequence length and 10% with 512 sequence length, we pre-train the model with 512 sequence length for 1M steps on a v3-8 TPU. |
|
|
|
The current version of the model is trained on Indo4B and small Twitter dump. |
|
|
|
## Acknowledgement |
|
|
|
Big thanks to TFRC (TensorFlow Research Cloud) for providing free TPU. |
|
|
