KordAI/III-BERT

IIIBERT Base MLM

IIIBERT is a compact Transformer language model developed for Thai text. This checkpoint was pretrained with a masked language modeling (MLM) objective and is intended as a starting point for downstream fine-tuning.

The model has approximately 28 million parameters. The model architecture and implementation are custom; the tokenizer is based on the pretrained SentencePiece tokenizer from airesearch/wangchanberta-base-wiki-spm.

Model details

• Model type: Custom Transformer encoder with a masked language modeling head
• Primary language: Thai
• Parameters: Approximately 28M
• Pretraining objective: Masked language modeling (MLM)
• Maximum sequence length used in pretraining: 512 tokens
• Tokenizer: airesearch/wangchanberta-base-wiki-spm (pretrained tokenizer; not trained from scratch for IIIBERT)
• Pretraining data: Thai Wikipedia and WangchanLION-Web
• License: Not specified. Check the licenses and terms of the model and training data before redistribution or commercial use.

Intended uses

IIIBERT Base MLM is intended to be fine-tuned for Thai NLP tasks, for example:

• Text classification, such as sentiment analysis
• Token classification, such as part-of-speech tagging and named entity recognition
• Other Thai language understanding tasks for which task-specific training data is available

The base MLM checkpoint is not itself a ready-made sentiment classifier, POS tagger, or NER system. Those tasks require a task-specific head and fine-tuning.

Load the model

The repository contains custom model code. Review that code before loading it and use trust_remote_code=True only if you trust the repository.

import torch
from transformers import CamembertTokenizer, AutoModelForMaskedLM

model_id = "KordAI/III-BERT"

tokenizer = CamembertTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
model.eval()

text = "ภาษาไทยเป็นภาษาที่น่าสนใจ"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
inputs = {key: value.to(device) for key, value in inputs.items()}

with torch.no_grad():
    outputs = model(**inputs)

print("Logits shape:", outputs.logits.shape)

Fill-mask example

Use the tokenizer's mask token to ask the MLM to score possible replacements. This is a demonstration of the pretraining objective, not a guarantee that the top prediction will be grammatically or factually correct.

import torch
from transformers import CamembertTokenizer, AutoModelForMaskedLM

model_id = "KordAI/III-BERT"
tokenizer = CamembertTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
model.eval()

text = f"บูดาเปสต์เป็นเมืองหลวงของ{tokenizer.mask_token}"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
inputs = {key: value.to(device) for key, value in inputs.items()}

mask_positions = (inputs["input_ids"][0] == tokenizer.mask_token_id).nonzero(as_tuple=True)[0]
if len(mask_positions) == 0:
    raise ValueError("The input does not contain the tokenizer's mask token.")

with torch.no_grad():
    logits = model(**inputs).logits

position = mask_positions[0].item()
top_ids = logits[0, position].topk(5).indices.tolist()

for token_id in top_ids:
    print(tokenizer.decode([token_id]), float(logits[0, position, token_id]))

Downstream task: sequence classification

For sequence classification, IIIBERT needs a classification head and task-specific fine-tuning. The model repository must expose IIIBERTForSequenceClassification through the Transformers AutoModelForSequenceClassification mapping. If the published config.json currently maps only AutoConfig, AutoModel, and AutoModelForMaskedLM, add the sequence-classification mapping in the upload script and republish before using the example below.

Example model loading for a four-class sentiment task:

from transformers import CamembertTokenizer, AutoModelForSequenceClassification

model_id = "KordAI/III-BERT"

label2id = {
    "negative": 0,
    "neutral": 1,
    "positive": 2,
    "question": 3,
}
id2label = {value: key for key, value in label2id.items()}

tokenizer = CamembertTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForSequenceClassification.from_pretrained(
    model_id,
    trust_remote_code=True,
    num_labels=len(label2id),
    label2id=label2id,
    id2label=id2label,
)

After loading, tokenize each example and train the model on a labeled dataset with the Hugging Face Trainer or a PyTorch training loop. Use the same label-to-ID mapping for training, validation, and inference. For token-level tasks such as POS and NER, use a token-classification head instead and align labels to the tokenizer's subwords.

Experiments

The following are results from downstream fine-tuning runs using IIIBERT. They are not evaluation scores for the untouched base MLM checkpoint. The scores below are reproduced from the reported test metrics for these experiments.

Thai sentiment classification

• Dataset: pythainlp/wisesight_sentiment
• Test loss: 0.7298
• Accuracy: 0.7252
• Macro F1: 0.6066
• Weighted F1: 0.7174
• F1 — negative: 0.7495
• F1 — neutral: 0.7812
• F1 — positive: 0.5180
• F1 — question: 0.3778

LST20 token-classification tasks

The following metrics were reported after five epochs of fine-tuning. Labels were propagated from each original word to all tokenizer subwords belonging to that word, and the reported evaluation was calculated at the subword level. These results should therefore not be compared directly with word-level or entity-span-level benchmark scores that use a different evaluation protocol.

Task	Test loss	Accuracy	Macro F1	Weighted F1
POS tagging	0.0947	0.9662	0.8546	0.9658
Clause tagging	0.0340	0.9871	0.9698	0.9871
Named entity recognition (NER)	0.1459	0.9550	0.7028	0.9546

The gap between macro F1 and weighted F1 for NER suggests uneven performance across labels, which is common when token labels are imbalanced. Per-class and entity-span evaluation should be examined before making stronger claims about NER quality.

Training information

• Pretraining data: Thai Wikipedia and WangchanLION-Web
• Tokenizer source: airesearch/wangchanberta-base-wiki-spm
• Maximum sequence length: 512
• Training objective: Masked language modeling
• Parameter count: Approximately 28M

Hardware, total training time, energy use, and pretraining hyperparameters are not reported here because verified values are not available in this model card.

Limitations

• Performance depends on the domain, label quality, and class distribution of downstream datasets.
• The tokenizer is pretrained separately; IIIBERT's encoder was trained with that tokenizer rather than with a tokenizer trained from scratch.
• MLM predictions are contextual token predictions and should not be treated as factual verification.
• Downstream results depend on the fine-tuning setup and evaluation protocol. The LST20 results above use subword-level metrics.
• The model may reproduce biases or errors present in its training data.
• Thai text from specialized domains, noisy text, code-switching, or text outside the training distribution may perform differently.

Responsible use

Evaluate the model on data representative of the intended application. For high-impact uses, use additional validation, human review, and appropriate safeguards. Do not treat model outputs as the sole basis for consequential decisions.

Citation

No paper or formal publication is associated with this model card at the time of writing. If the model is used in research or an application, cite the Hugging Face model repository and describe the model version and evaluation setup.