vn-zh-nmt

A bidirectional Vietnamese-Chinese Neural Machine Translation model trained with curriculum learning and RoPE attention.

Model Details

• Model Type: Vietnamese-Chinese Translation
• Architecture: Transformer with Grouped Query Attention (GQA) and RoPE
• Language Pair: Vietnamese ↔ Chinese (Simplified)
• Training Framework: PyTorch
• License: MIT

Training Metrics

• Training Loss: 0.19108145256148706
• Validation Loss: 1.1668201344353812
• BLEU Score: 50.62326470643003
• Epoch: 21
• Learning Rate: 1.1942420399468553e-05

Usage

import torch
from model.transformer_model import TransformerModel

# Load checkpoint
checkpoint = torch.load('model.pt', map_location='cuda')
model = TransformerModel(checkpoint['config'], vocab_size=8000)
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()

# Inference (tokenization and decoding depend on SentencePiece)
# See training/utils.py for greedy_decode and beam_search_decode

Model Architecture

Key Components

• Encoder Layers: 8
• Decoder Layers: 8
• Hidden Dimension: 768
• Attention Heads: 12
• KV Heads (GQA): 4
• Feed-Forward Dimension: 3072
• Max Sequence Length: 32
• Vocabulary Size: 8000

Special Tokens

• <pad>: Padding token
• <s>: Begin-of-sequence
• </s>: End-of-sequence
• <2zh>: Prefix for Vietnamese→Chinese
• <2vi>: Prefix for Chinese→Vietnamese

Training Details

Data

• Source: Vietnamese-Chinese parallel corpus
• Train/Val Split: 95%/5%
• Token-length Filtering: Max 32 tokens

Curriculum Learning

• zh→vi: Full coverage every epoch
• vi→zh: Progressive window covering 70% per epoch
• Span Masking: 1% probability for robustness

Optimization

• Optimizer: AdamW (β₁=0.9, β₂=0.98, ε=1e-9)
• Learning Rate Schedule: Warmup + Inverse Sqrt Decay
• Warmup Steps: 200
• Batch Size: 128
• Label Smoothing: 0.01
• Gradient Clipping: 1.0

Citation

If you use this model, please cite:

@misc{vn_zh_nmt,
  title={Vietnamese-Chinese Neural Machine Translation},
  author={Duy},
  year={2026}
}

License

This project is licensed under the MIT License.