KordAI/III-BERT-EN

IIIBERT Base MLM

IIIBERT is a compact Transformer language model developed for English text. This checkpoint was pretrained with a masked language modeling (MLM) objective and is intended as a starting point for downstream fine-tuning.

The model has approximately 32 million parameters. The model architecture and implementation are custom; the tokenizer is the pretrained byte-level BPE tokenizer from FacebookAI/roberta-base.

Model details

• Model type: Custom Transformer encoder with a masked language modeling head
• Primary language: English
• Parameters: Approximately 32M
• Pretraining objective: Masked language modeling (MLM), 15% masking probability
• Maximum sequence length used in pretraining: 512 tokens
• Tokenizer: FacebookAI/roberta-base (pretrained tokenizer; not trained from scratch for IIIBERT)
• Pretraining data: English Wikipedia (wikimedia/wikipedia, 20231101.en) and C4 (allenai/c4, en), mixed at roughly 30% / 70%
• License: Not specified. Check the licenses and terms of the model and training data before redistribution or commercial use.

Intended uses

IIIBERT Base MLM is intended to be fine-tuned for English NLP tasks, for example:

• Text classification, such as sentiment analysis
• Token classification, such as part-of-speech tagging and named entity recognition
• Other English language understanding tasks for which task-specific training data is available

The base MLM checkpoint is not itself a ready-made sentiment classifier, POS tagger, or NER system. Those tasks require a task-specific head and fine-tuning.

Load the model

The repository contains custom model code. Review that code before loading it and use trust_remote_code=True only if you trust the repository.

import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM

model_id = "KordAI/III-BERT-EN"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
model.eval()

text = "Language models are useful for many tasks."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
inputs = {key: value.to(device) for key, value in inputs.items()}

with torch.no_grad():
    outputs = model(**inputs)

print("Logits shape:", outputs.logits.shape)

Fill-mask example

Use the tokenizer's mask token to ask the MLM to score possible replacements. This is a demonstration of the pretraining objective, not a guarantee that the top prediction will be grammatically or factually correct.

import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM

model_id = "KordAI/III-BERT-EN"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
model.eval()

text = f"Budapest is the capital of {tokenizer.mask_token}."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
inputs = {key: value.to(device) for key, value in inputs.items()}

mask_positions = (inputs["input_ids"][0] == tokenizer.mask_token_id).nonzero(as_tuple=True)[0]
if len(mask_positions) == 0:
    raise ValueError("The input does not contain the tokenizer's mask token.")

with torch.no_grad():
    logits = model(**inputs).logits

position = mask_positions[0].item()
top_ids = logits[0, position].topk(5).indices.tolist()

for token_id in top_ids:
    print(tokenizer.decode([token_id]), float(logits[0, position, token_id]))

Downstream task: sequence classification

For sequence classification, IIIBERT needs a classification head and task-specific fine-tuning. The model repository must expose IIIBERTForSequenceClassification through the Transformers AutoModelForSequenceClassification mapping. If the published config.json currently maps only AutoConfig, AutoModel, and AutoModelForMaskedLM, add the sequence-classification mapping in the upload script and republish before using the example below.

Example model loading for a three-class sentiment task:

from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "KordAI/III-BERT-EN"

label2id = {
    "negative": 0,
    "neutral": 1,
    "positive": 2,
}
id2label = {value: key for key, value in label2id.items()}

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(
    model_id,
    trust_remote_code=True,
    num_labels=len(label2id),
    label2id=label2id,
    id2label=id2label,
)

After loading, tokenize each example and train the model on a labeled dataset with the Hugging Face Trainer or a PyTorch training loop. Use the same label-to-ID mapping for training, validation, and inference. For token-level tasks such as POS and NER, use a token-classification head instead and align labels to the tokenizer's subwords.

Training information

• Pretraining data: English Wikipedia (20231101.en) and C4 (en)
• Tokenizer source: FacebookAI/roberta-base
• Maximum sequence length: 512
• Training objective: Masked language modeling
• Parameter count: Approximately [N]M

Hardware, total training time, energy use, and evaluation results are not reported here because verified values are not available in this model card.

Limitations

• Performance depends on the domain, label quality, and class distribution of downstream datasets.
• The tokenizer is pretrained separately; IIIBERT's encoder was trained with that tokenizer rather than with a tokenizer trained from scratch.
• MLM predictions are contextual token predictions and should not be treated as factual verification.
• No evaluation results are provided for the base checkpoint, so downstream performance should be measured on your own task.
• The model may reproduce biases or errors present in its training data, which includes web-crawled text.
• Text from specialized domains, noisy text, code-switching, or text outside the training distribution may perform differently.

Responsible use

Evaluate the model on data representative of the intended application. For high-impact uses, use additional validation, human review, and appropriate safeguards. Do not treat model outputs as the sole basis for consequential decisions.

Citation

No paper or formal publication is associated with this model card at the time of writing. If the model is used in research or an application, cite the Hugging Face model repository and describe the model version and evaluation setup.