---
license: bsd-3-clause-clear
language:
- ne
metrics:
- perplexity
library_name: transformers
pipeline_tag: text-generation
---

# NepaliGPT: Nepali Language Generative Pretrained Transformer Model
This is an experiment for developing a language generation model for the Nepali language. 
Causal Language Model which can predict the next possible tokens given a context in Nepali language. 

# Dataset Used
A large corpus of 9.3 GB size has been collected from different sources on the internet. The sources include
- Nepali Books found online.
- Nepali News Article from Nepali news portals.
- Nepali text collected from different open source Nepali NLP datasets. 

# Hyperparameters Used
Learning rate -> 2e-5 \
Weight Decay -> 0.01 \
Number of training epochs -> 5 \ 
bf16 -> True \
Base Model Architecture -> GPT-2 \

## Training Results

It achieves the following results on the evaluation set:

| Training Loss | Validation Loss | Perplexity
|:-------------:|:---------------:|:----------:|
| 3.3968        | 3.2705          | 26.3245