NIVED2003/gemma-4-E4B-dora-prose-half1

Gemma-4-E4B DoRA: Sanskrit $\leftrightarrow$ Hindi Contemporary Prose (Half 1)

This repository contains a DoRA (Weight-Decomposed Low-Rank Adaptation) adapter for google/gemma-4-E4B fine-tuned for bidirectional translation between Contemporary Sanskrit and Modern Hindi.

This model represents Half 1 of a dual-half data split experiment designed to study within-domain adaptation stability, magnitude/direction dynamics ($\Delta M$ vs $\Delta D$), and representation divergence against classical poetry translation.

---

Model Details

• Base Model: google/gemma-4-E4B
• Tuning Method: DoRA (Weight-Decomposed Low-Rank Adaptation)
• Rank ($r$): 16
• Alpha ($\alpha$): 32 ($\alpha / r = 2.0$)
• LoRA Dropout: 0.05
• Target Modules: All 7 linear projection layers in the 42 text decoder layers:
  • q_proj, k_proj, v_proj, o_proj (Self-Attention)
  • gate_proj, up_proj, down_proj (MLP)
• Trainable Parameters: 37,925,888 / 8,034,082,336 (0.472%)
• Language Pair: Sanskrit (sa) $\leftrightarrow$ Hindi (hi) (Bidirectional)
• Domain: Contemporary Sanskrit Prose & Parallel Benchmarks (Samanantar, Flores, IN22)

---

Evaluation & Training Dynamics

The model was trained for 2 epochs on 169,994 parallel sentence pairs and evaluated on 2,000 held-out validation pairs.

Validation Loss & Perplexity Trajectory

Milestone	Checkpoint Step	Epoch	Eval Loss (Cross-Entropy)	Perplexity ($\text{PPL} = e^{\text{loss}}$)
Milestone 25%	5,312	0.50	1.561	4.76
Milestone 50%	10,625	1.00	1.411	4.10
Milestone 75%	15,938	1.50	1.347	3.85
Final (100%)	21,250	2.00	1.326	3.77

The validation loss decreased monotonically across all four checkpoints down to 1.326 (Perplexity 3.77), indicating high translation fluency with sharp convergence.

Within-Domain Stability (Prose Half 1 vs. Prose Half 2)

Comparing both halves of the independent prose splits confirms high empirical consistency:

Metric	Prose Half 1	Prose Half 2	Absolute Delta ($\Delta$)	Relative Variance
Validation Loss	1.326	1.354	0.028	< 2.1%
Validation Perplexity	3.77	3.87	0.10	< 2.6%
Final Train Loss (Smoothed)	5.505	5.352	0.153	< 2.8%

---

Training Hyperparameters

• Hardware: 1x NVIDIA H100 80GB SXM5 GPU (Sapphire Rapids host)
• Runtime: 10 hours 10 minutes (36,610s)
• Throughput: 9.29 samples/sec (0.58 optimizer steps/sec)
• Effective Batch Size: 16 (per_device_batch_size=4, gradient_accumulation_steps=4)
• Optimizer: AdamW (weight_decay=0.01, max_grad_norm=1.0)
• Learning Rate: 5e-5 with Cosine Annealing and 100 linear warmup steps
• Sequence Length: 384 tokens (loss computed strictly on target tokens with -100 prompt masking)
• Precision: bfloat16

---

How to Use

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

BASE_MODEL = "google/gemma-4-E4B"
ADAPTER_REPO = "NIVED2003/gemma-4-E4B-dora-prose-half1"

# 1. Load Tokenizer & Base Model
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# 2. Load DoRA Adapter
model = PeftModel.from_pretrained(model, ADAPTER_REPO)
model.eval()

# 3. Sanskrit -> Hindi Translation
sanskrit_text = "भारतं विश्वस्य प्राचीनतमासु संस्कृतिषु अन्यतमा वर्तते।"
prompt = f"Instruction: Translate the following contemporary Sanskrit text to Hindi.\nInput: {sanskrit_text}\nOutput: "

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=128,
        temperature=0.3,
        top_p=0.9,
        do_sample=True,
        eos_token_id=tokenizer.eos_token_id
    )

translation = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print("Hindi Translation:", translation)

# 4. Hindi -> Sanskrit Translation
hindi_text = "विज्ञान और तकनीक के क्षेत्र में नए शोध अत्यंत महत्वपूर्ण हैं।"
prompt_reverse = f"Instruction: Translate the following contemporary Hindi text to Sanskrit.\nInput: {hindi_text}\nOutput: "

inputs_rev = tokenizer(prompt_reverse, return_tensors="pt").to("cuda")
with torch.no_grad():
    outputs_rev = model.generate(
        **inputs_rev,
        max_new_tokens=128,
        temperature=0.3,
        top_p=0.9,
        do_sample=True,
        eos_token_id=tokenizer.eos_token_id
    )

translation_sa = tokenizer.decode(outputs_rev[0][inputs_rev.input_ids.shape[1]:], skip_special_tokens=True)
print("Sanskrit Translation:", translation_sa)

---

Intermediate Milestones Available

All intermediate milestone checkpoints are versioned in this repository:

• checkpoints/step_25pct: 25% training milestone (step 5,312)
• checkpoints/step_50pct: 50% training milestone (step 10,625)
• checkpoints/step_75pct: 75% training milestone (step 15,938)
• Root directory (.): Final fully-trained 100% adapter (step 21,250)

---

Citation & Architecture

@inproceedings{liu2024dora,
  title={DoRA: Weight-Decomposed Low-Rank Adaptation},
  author={Liu, Shih-Yang and Wang, Chien-Yi and Yin, Hongxu and Khona, Pavlo and Shen, Sheng and Yen, Chen-Yu and Wang, Ting-Kuei and Chen, Kuan-Yu and Darve, Eric and Chen, Kwang-Ting},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2024}
}