Edit model card
YAML Metadata Warning: empty or missing yaml metadata in repo card (https://huggingface.co/docs/hub/model-cards#model-card-metadata)

inference: false language: id

IndoConvBERT Base Model

IndoConvBERT is a ConvBERT model pretrained on Indo4B.

Pretraining details

We follow a different training procedure: instead of using a two-phase approach, that pre-trains the model for 90% with 128 sequence length and 10% with 512 sequence length, we pre-train the model with 512 sequence length for 1M steps on a v3-8 TPU.

The current version of the model is trained on Indo4B and small Twitter dump.


Big thanks to TFRC (TensorFlow Research Cloud) for providing free TPU.

Downloads last month
Hosted inference API
This model can be loaded on the Inference API on-demand.