PaxiAI/Vietnamese-Encoder-Base-v1

PaxiAI Vietnamese Encoder Base v1

PaxiAI Vietnamese Encoder Base v1 is a Vietnamese-focused bidirectional Transformer encoder pretrained from scratch using Masked Language Modeling (MLM). It uses the architecture principles of ModernBERT, all model parameters trained from scratch with the PaxiAI Vietnamese Tokenizer.

This model is intended to serve as a reusable foundation encoder for Vietnamese NLP tasks such as:

• semantic embeddings
• information retrieval
• reranking
• text classification
• intent detection
• named entity recognition
• semantic similarity
• sentence representation
• domain-specific encoder models

---

Model Overview

Property	Value
Architecture	Bidirectional Transformer Encoder
Training objective	Masked Language Modeling
Initialization	Random weights
Hidden size	768
Transformer layers	12
Attention heads	12
Intermediate size	2048
Vocabulary size	48,000
Maximum sequence length	2,048
Primary language	Vietnamese
Tokenizer	PaxiAI/Vietnamese-Tokenizer
Pretraining steps	200,000
Pretrained model dependency	None

---

Tokenizer

The model uses:

PaxiAI/Vietnamese-Tokenizer

https://huggingface.co/PaxiAI/Vietnamese-Tokenizer

The tokenizer has a vocabulary size of approximately 48K tokens and was designed primarily for Vietnamese text while retaining support for English and programming-related text.

For MLM pretraining, one of the tokenizer's reserved tokens is used as the mask token without modifying the original vocabulary or token ID mapping.

---

Loading the Model

from transformers import AutoTokenizer, AutoModel

model_id = "PaxiAI/Vietnamese-Encoder-Base-v1"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id)

text = "Trí tuệ nhân tạo đang phát triển rất nhanh."

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=512,
)

outputs = model(**inputs)

print(outputs.last_hidden_state.shape)

The output contains contextual token representations:

[batch_size, sequence_length, 768]

License

Please refer to the license associated with this repository and verify the licenses and usage conditions of the individual datasets used during pretraining.