my_awesome_eli5_clm-model

A Causal Language Model fine-tuned on Arabic text, based on distilbert/distilgpt2.
Fine-tuned by EhabSuliman as part of an LLM course project.

Model Description

• Model type: Causal Language Model (CLM)
• Base model: distilbert/distilgpt2
• Language: English
• Fine-tuned by: EhabSuliman
• License: Apache 2.0

Intended Uses & Limitations

Intended uses:

• Arabic text generation
• Language modeling research
• Educational/learning purposes

Limitations:

• Trained on a relatively small dataset
• Loss is still relatively high (3.8027), meaning the model may generate inaccurate or repetitive text
• Not recommended for production use without further fine-tuning

How to Use

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("EhabSuliman/my_awesome_eli5_clm-model")
model = AutoModelForCausalLM.from_pretrained("EhabSuliman/my_awesome_eli5_clm-model")

prompt = "Somatic hypermutation allows the immune system to"
inputs = tokenizer(prompt, return_tensors="pt").input_ids
outputs = model.generate(
    inputs,
    max_new_tokens=100,
    do_sample=True,
    top_k=50,
    top_p=0.95
)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

Training and Evaluation Data

Fine-tuned on an Arabic text dataset using the ELI5 (Explain Like I'm 5) format.

Training Procedure

Training Hyperparameters

The following hyperparameters were used during training:

• Learning rate: 2e-05
• Train batch size: 8
• Eval batch size: 8
• Seed: 42
• Optimizer: AdamW (fused) with betas=(0.9, 0.999), epsilon=1e-08
• LR scheduler: Linear
• Epochs: 3

Training Results

Training Loss	Epoch	Step	Validation Loss
3.8556	1.0	1327	3.8101
3.7851	2.0	2654	3.8035
3.7514	3.0	3981	3.8027

The model shows steady improvement across epochs with validation loss decreasing from 3.8101 → 3.8027.

Framework Versions

• Transformers 5.9.0
• PyTorch 2.11.0+cu128
• Datasets 4.0.0
• Tokenizers 0.22.2

Author

Ehab Suliman — Machine Learning Engineer
🔗 HuggingFace Profile