AIOKiet/nllb-base-iwslt2015-en-vi

NLLB-200-Distilled-600M Fine-tuned for English-to-Vietnamese Translation

This model is a fine-tuned version of facebook/nllb-200-distilled-600M for English-to-Vietnamese neural machine translation.

The model was fine-tuned exclusively on the IWSLT2015 English-Vietnamese dataset as part of a controlled comparison with mBART-50 and EnViT5-Base.

No additional parallel corpus, back-translation data, synthetic data, or data augmentation was used in this baseline.

Model Details

• Base model: facebook/nllb-200-distilled-600M
• Model family: NLLB-200
• Variant: Distilled 600M
• Task: English-to-Vietnamese machine translation
• Architecture: Encoder-decoder Transformer
• Source language: English
• Target language: Vietnamese
• NLLB source language code: eng_Latn
• NLLB target language code: vie_Latn
• Framework: Hugging Face Transformers
• Training hardware: 1 × NVIDIA Tesla T4
• Training precision: FP16
• Training epochs: 2
• Training steps: 33,330
• Random seed: 42
• Checkpoint in this repository: Final checkpoint after 2 epochs
• License: CC-BY-NC-4.0

The base NLLB checkpoint is implemented in Hugging Face Transformers through the M2M100 conditional generation architecture. Therefore, loading the model may display the class name M2M100ForConditionalGeneration. This is expected for this NLLB checkpoint.

Model Source

• Base model: https://huggingface.co/facebook/nllb-200-distilled-600M
• NLLB paper: https://arxiv.org/abs/2207.04672
• Fine-tuning dataset: https://huggingface.co/datasets/nguyenvuhuy/iwslt2015-en-vi

Training Data

The model was fine-tuned using:

nguyenvuhuy/iwslt2015-en-vi

Dataset splits:

Split	Sentence pairs
Train	133,317
Validation	1,268
Test	1,268

Only the IWSLT2015 English-Vietnamese data was used for fine-tuning.

No additional training corpus was introduced.

Preprocessing

The NLLB tokenizer was configured with:

Source language: eng_Latn
Target language: vie_Latn

The tokenizer was initialized using:

tokenizer = AutoTokenizer.from_pretrained(
    "facebook/nllb-200-distilled-600M",
    src_lang="eng_Latn",
    tgt_lang="vie_Latn"
)

The Vietnamese language token was used as the forced beginning-of-sequence token during generation:

TARGET_LANG_ID = tokenizer.convert_tokens_to_ids("vie_Latn")

model.generation_config.forced_bos_token_id = TARGET_LANG_ID

Maximum sequence lengths:

Parameter	Value
Maximum source length	128
Maximum target length	128

Target padding token IDs were replaced with -100 so that padding positions were ignored when computing the training loss.

No additional text normalization, corpus filtering, back-translation, or data augmentation was applied.

Training Hyperparameters

Hyperparameter	Value
Training epochs	2
Training steps	33,330
Training batch size	8
Evaluation batch size	8
Gradient accumulation steps	1
Effective batch size	8
Learning rate	5e-5
Optimizer	AdamW
Weight decay	0.01
Learning-rate scheduler	Linear
Warmup ratio	0.0
Maximum gradient norm	1.0
Mixed precision	FP16
BF16	Disabled
Gradient checkpointing	Enabled
Random seed	42
Data seed	42
Evaluation interval	5,000 steps
Maximum source length	128
Maximum target length	128
Beam size	5
Maximum generation length	128
Evaluation accumulation steps	8

Training Runtime

The complete two-epoch fine-tuning run produced:

Statistic	Value
Global steps	33,330
Epochs	2
Training runtime	23,588.53 seconds
Training time	approximately 6 h 33 min 9 s
Samples per second	11.304
Steps per second	1.413
Final reported mean training loss	1.376647
Total FLOPs	7.2228 × 10^16

The experiment was performed using a single NVIDIA Tesla T4 GPU.

Validation Results

Validation performance observed during training:

Step	Training Loss	Validation Loss	BLEU ↑	chrF++ ↑	TER ↓
5,000	1.5013	1.387950	34.9342	54.1157	45.9108
10,000	1.4481	1.356183	35.3538	54.2255	45.6079
15,000	1.4088	1.335958	35.2021	54.6027	45.7682
20,000	1.2878	1.330488	35.3907	54.5508	45.6109
25,000	1.2803	1.316011	35.6443	54.6142	45.4060
30,000	1.2746	1.306112	36.0011	55.1273	45.2189

Among the scheduled validation evaluations:

• Highest BLEU: 36.0011 at step 30,000
• Highest chrF++: 55.1273 at step 30,000
• Lowest TER: 45.2189 at step 30,000
• Lowest validation loss: 1.306112 at step 30,000

All three translation metrics continued to improve overall toward the end of training.

The model reached its strongest scheduled validation result at step 30,000.

Checkpoint Selection

This experiment used a fixed training budget of two epochs.

Checkpoint saving was disabled during this experimental run. Therefore, the model stored in this repository corresponds to the final model after 33,330 optimization steps.

The scheduled validation evaluations were performed every 5,000 steps, with the final scheduled validation measurement occurring at step 30,000.

The checkpoint at step 30,000 was not retained separately.

For the controlled baseline comparison, the final two-epoch checkpoint is used consistently across NLLB-200-Distilled-600M, mBART-50, and EnViT5-Base.

Evaluation

Evaluation Dataset

The final model is evaluated on the IWSLT2015 English-Vietnamese test split:

Split	Samples
Test	1,268

The test set is not used for model selection.

Decoding Configuration

Parameter	Value
Beam search	5 beams
Maximum generation length	128
Target language	vie_Latn
Forced target language token	Enabled

Evaluation Metrics

The following metrics are used:

• BLEU: Higher is better
• chrF++: Higher is better
• TER: Lower is better
• COMET: Higher is better

BLEU, chrF++, and TER are computed using SacreBLEU.

COMET is evaluated separately for the final comparison of the baseline models.

Test Results

Final test-set results:

Metric	Score
BLEU ↑	TBD
chrF++ ↑	TBD
TER ↓	TBD
COMET ↑	TBD

The values above should be updated after evaluating the final checkpoint on the complete 1,268-sentence IWSLT2015 test set.

SacreBLEU Tokenization Warning

During validation, SacreBLEU may report a warning similar to:

It looks like you forgot to detokenize your test data.

The IWSLT2015 dataset used in this experiment contains punctuation patterns that SacreBLEU detects as potentially pre-tokenized text.

For the controlled baseline comparison, the same dataset representation and evaluation procedure are retained for all compared models.

The warning does not indicate a training failure.

No model-specific detokenization is applied because doing so for only one baseline would make the evaluation protocol inconsistent.

How to Use

import torch

from transformers import AutoTokenizer
from transformers import AutoModelForSeq2SeqLM

model_name = "AIOKiet/nllb-200-distilled-600m-iwslt2015-en-vi"

SRC_LANG = "eng_Latn"
TGT_LANG = "vie_Latn"

tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    src_lang=SRC_LANG,
    tgt_lang=TGT_LANG
)

model = AutoModelForSeq2SeqLM.from_pretrained(
    model_name
)

device = torch.device(
    "cuda" if torch.cuda.is_available() else "cpu"
)

model = model.to(device)
model.eval()

target_lang_id = tokenizer.convert_tokens_to_ids(
    TGT_LANG
)

text = "Artificial intelligence is changing the world."

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=128
).to(device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        forced_bos_token_id=target_lang_id,
        num_beams=5,
        max_length=128
    )

translation = tokenizer.batch_decode(
    outputs,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False
)[0]

print(translation)

Example usage:

Input:
Artificial intelligence is changing the world.

Output:
<English-to-Vietnamese translation>

Intended Use

This fine-tuned model is intended for:

• English-to-Vietnamese machine translation research
• Evaluation on IWSLT2015
• Controlled neural machine translation baseline experiments
• Comparison with mBART-50 and EnViT5-Base
• Research on parameter-efficient fine-tuning methods
• Experimental English-to-Vietnamese translation applications

Downstream Use

The model may be used as a research component in applications such as:

• Translation demonstrations
• Experimental translation web applications
• Translation APIs
• Comparative NMT studies
• Parameter-efficient fine-tuning experiments

Users should evaluate the model on their own target domain before using it outside the IWSLT2015 experimental setting.

Out-of-Scope Use

The model has not been validated for:

• Certified professional translation
• Medical translation
• Legal translation
• Safety-critical translation
• High-stakes automated decision making
• Unsupervised production deployment

The original NLLB-200 model is released primarily for machine translation research rather than unrestricted production deployment.

Limitations

This fine-tuned checkpoint has several important limitations.

Domain Limitation

The model was fine-tuned only on IWSLT2015 English-Vietnamese data.

The corpus primarily represents TED-talk-style language, so performance may decrease substantially for text from different domains.

Language Direction

This fine-tuned checkpoint was trained specifically for:

English → Vietnamese

Performance for Vietnamese-to-English or other NLLB language pairs was not evaluated as part of this experiment.

Sequence Length

Training and evaluation use a maximum sequence length of 128 tokens.

Longer text may be truncated and should normally be segmented before translation.

Checkpoint Limitation

The repository contains the final checkpoint at step 33,330 rather than a separately preserved best validation checkpoint.

Evaluation Limitation

Automatic metrics such as BLEU, chrF++, TER, and COMET do not fully capture translation adequacy, fluency, terminology, factuality, or human preference.

Human evaluation is recommended for more comprehensive assessment.

Bias and Risks

The fine-tuned model inherits limitations and potential biases from:

1. The original NLLB-200 training data
2. The IWSLT2015 English-Vietnamese fine-tuning data
3. Automatic data processing and tokenization procedures
4. The limited domain represented by the fine-tuning corpus

Generated translations should not automatically be assumed to be accurate, neutral, or suitable for high-stakes use.

Experimental Context

This model is one of three controlled English-to-Vietnamese baselines:

Model	Role
mBART-50	Multilingual sequence-to-sequence baseline
NLLB-200-Distilled-600M	Multilingual machine translation baseline
EnViT5-Base	English-Vietnamese specialized baseline

The baseline experiments use the same:

• IWSLT2015 training, validation, and test splits
• Number of training epochs
• Batch size
• Gradient accumulation
• Learning rate
• Optimizer
• Weight decay
• Maximum source length
• Maximum target length
• FP16 training
• Evaluation interval
• Beam size
• Maximum generation length
• Random seed
• Test set
• Evaluation metrics

Model-specific language-token handling is preserved according to the requirements of each pretrained architecture.

For NLLB, eng_Latn and vie_Latn language identifiers are used.

Reproducibility

Main configuration:

Base model:
facebook/nllb-200-distilled-600M

Dataset:
nguyenvuhuy/iwslt2015-en-vi

Direction:
English -> Vietnamese

Source language code:
eng_Latn

Target language code:
vie_Latn

Train samples:
133317

Validation samples:
1268

Test samples:
1268

Epochs:
2

Training steps:
33330

Batch size:
8

Gradient accumulation:
1

Effective batch size:
8

Learning rate:
5e-5

Optimizer:
AdamW

Weight decay:
0.01

LR scheduler:
Linear

Warmup ratio:
0.0

Maximum gradient norm:
1.0

Maximum source length:
128

Maximum target length:
128

FP16:
True

BF16:
False

Gradient checkpointing:
True

Evaluation interval:
5000 steps

Beam size:
5

Maximum generation length:
128

Seed:
42

Data seed:
42

Compute Infrastructure

Hardware

• GPU: NVIDIA Tesla T4
• Number of GPUs: 1
• Precision: FP16

Runtime

Training runtime: 23588.5289 seconds
Approximate duration: 6 h 33 min 9 s
Training samples/second: 11.304
Training steps/second: 1.413

Technical Details

The original checkpoint is:

facebook/nllb-200-distilled-600M

NLLB is a multilingual encoder-decoder neural machine translation model.

For this fine-tuning experiment, the pretrained multilingual model is specialized to English-to-Vietnamese translation using IWSLT2015.

During generation, Vietnamese is explicitly selected using the NLLB language identifier:

vie_Latn

and its corresponding token ID is supplied as the forced beginning-of-sequence token.

License

The base facebook/nllb-200-distilled-600M checkpoint is distributed under:

CC-BY-NC-4.0

Users of this fine-tuned model should comply with the license and usage conditions of the original NLLB checkpoint.

Citation

If you use this model, please cite the original NLLB work:

@article{nllb2022,
  title={No Language Left Behind: Scaling Human-Centered Machine Translation},
  author={{NLLB Team} and others},
  journal={arXiv preprint arXiv:2207.04672},
  year={2022}
}

A citation for the associated English-Vietnamese fine-tuning study will be added if the study is published.

Model Card Author

Fine-tuned and documented for controlled English-to-Vietnamese neural machine translation experiments using NLLB-200-Distilled-600M and IWSLT2015.