Adc05102002/bge-m3-vi-base

BGE-M3 Custom Tokenizer (8.5K Vocab)

A customized version of :contentReference[oaicite:0]{index=0} with a newly trained tokenizer optimized for domain-specific multilingual retrieval workloads.

This model replaces the original XLM-R tokenizer vocabulary with a compact 8.5K-token tokenizer trained on a custom corpus.

Highlights

• Based on BAAI/bge-m3
• Custom tokenizer trained from scratch
• Reduced vocabulary size: 8500
• Long-context support: 8192 tokens
• Multilingual retrieval and embedding model
• Optimized for:
  • semantic search
  • RAG pipelines
  • dense retrieval
  • domain-specific embeddings

---

Model Details

Base Model

• Architecture: XLM-RoBERTa
• Original model: BAAI/bge-m3
• Embedding dimension: 1024
• Transformer encoder model

Tokenizer

The original tokenizer was replaced with a newly trained tokenizer using:

tokenizer = base_tokenizer.train_new_from_iterator(
    batch_iterator(),
    vocab_size=8500,
    min_frequency=2,
)