Librarian Bot: Update dataset YAML metadata for model (#1)

6b6c93f over 1 year ago

No virus

6.67 kB

	---
	license: apache-2.0
	tags:
	- generated_from_trainer
	datasets: Sakonii/nepalitext-language-model-dataset
	widget:
	- text: नेपाल र भारतबीच
	example_title: Example 1
	- text: प्रधानमन्त्री
	example_title: Example 2
	- text: 'दस वर्ष लामो '
	example_title: Example 3
	- text: 'जापानमा आज '
	example_title: Example 4
	- text: नेपालका धेरैजसो चाडपर्वहरूमध्ये,
	example_title: Example 5
	model-index:
	- name: distilgpt2-nepali
	results: []
	---

	# distilgpt2-nepali

	This model is pre-trained on [nepalitext](https://huggingface.co/datasets/Sakonii/nepalitext-language-model-dataset) dataset consisting of over 13 million Nepali text sequences using a Causal language modeling (CLM) objective. Our approach trains a Sentence Piece Model (SPM) for text tokenization similar to [XLM-ROBERTa](https://arxiv.org/abs/1911.02116) and trains [distilgpt2](https://huggingface.co/distilgpt2) for language modeling.

	It achieves the following results on the evaluation set:

	\| Training Loss \| Validation Loss \| Perplexity
	\|:-------------:\|:---------------:\|:----------:\|
	\| 3.3968 \| 3.2705 \| 26.3245

	## Model description

	Refer to original [distilgpt2](https://huggingface.co/distilgpt2)

	## Intended uses & limitations

	This raw model can be used for Nepali text generation and intends to be fine-tuned on Nepali language focused downstream task.
	The language model being trained on a data with texts grouped to a block size of 512, it handles text sequence up to 512 tokens and may not perform satisfactorily on shorter sequences.

	## Usage

	This model can be used directly with a pipeline for text generation. Since the generation relies on some randomness, we set a seed for reproducibility:

	```python
	>>> from transformers import pipeline, set_seed
	>>> set_seed(42)
	>>> generator = pipeline('text-generation', model='Sakonii/distilgpt2-nepali')
	>>> generator("नेपालका धेरैजसो चाडपर्वहरूमध्ये,", max_length=30, num_return_sequences=5)

	Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
	[{'generated_text': 'नेपालका धेरैजसो चाडपर्वहरूमध्ये, तिहार र छठपर्व विशेष रूपमा मनाइने भएकाले नेपाली मौलिक पर्व पनि हो । हिन्दू धर्म र संस्कृतिक... काठमाडौं ।'},
	{'generated_text': 'नेपालका धेरैजसो चाडपर्वहरूमध्ये, तिहारको मुख्य दिन आज साँझ अस्ताउँदो सूर्यलाई अर्घ्य दिइएको छ । वैदिक विधि...विस्तृतमा पढ्नुस् काठमाडौं । नेपाल चिकित्सक संघका'},
	{'generated_text': 'नेपालका धेरैजसो चाडपर्वहरूमध्ये, चाडपर्व, विवाह,... नेपाली काँग्रेसका प्रवक्ता विश्वप्रकाश शर्माले पार्टीभित्र आन्तरिक झगडा हुने निश्चित भएको र गुटबन्दीका कारण चुनावमा हार बेहोर्नु'},
	{'generated_text': 'नेपालका धेरैजसो चाडपर्वहरूमध्ये, दशैं नेपालीहरूको मौलिक पर्वका रूपमा मनाउँछन् । नेपालीहरूको दोस्रो महान् पर्व तिहार हो । तिहारले दाजुभाइ तथा दिदीबहिनीहरूको बीचमा प्रगाढ सम्बन्ध स्थापित'},
	{'generated_text': 'नेपालका धेरैजसो चाडपर्वहरूमध्ये, माघे संक्रान्ति र माघे संक्रान्तिमा माघे संक्रान्तिमा मात्र नभएर फागुन महिनाभर नै विशेष महत्व रहने गरेको छ । काठमाडौं ।'}]
	```

	Here is how we can use the model to get the features of a given text in PyTorch:

	```python
	from transformers import AutoTokenizer, AutoModelForCausalLM

	tokenizer = AutoTokenizer.from_pretrained('Sakonii/distilgpt2-nepali')
	model = AutoModelForCausalLM.from_pretrained('Sakonii/distilgpt2-nepali')

	# prepare input
	text = "चाहिएको text यता राख्नु होला।"
	encoded_input = tokenizer(text, return_tensors='pt')

	# forward pass
	output = model(**encoded_input)
	```

	## Training data

	This model is trained on [nepalitext](https://huggingface.co/datasets/Sakonii/nepalitext-language-model-dataset) language modeling dataset which combines the datasets: [OSCAR](https://huggingface.co/datasets/oscar) , [cc100](https://huggingface.co/datasets/cc100) and a set of scraped Nepali articles on Wikipedia.
	As for training the language model, the texts are tokenized using Sentence Piece Model (SPM), a vocabulary size of 24,576 and texts are are grouped to a block of 512 tokens.

	## Training procedure

	The model is trained with the same configuration as the original [distilgpt2](https://huggingface.co/distilgpt2); but with 512 tokens per instance, 12 instances per batch, and around 188.8K training steps.


	### Training hyperparameters

	The following hyperparameters were used during training:
	- learning_rate: 5e-05
	- train_batch_size: 12
	- eval_batch_size: 12
	- seed: 42
	- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
	- lr_scheduler_type: linear
	- num_epochs: 5
	- mixed_precision_training: Native AMP

	### Training results

	\| Training Loss \| Epoch \| Step \| Validation Loss \| Perplexity \|
	\|:-------------:\|:-----:\|:------:\|:---------------:\|:----------:\|
	\| 3.7645 \| 1.0 \| 94395 \| 3.6291 \| 37.6789 \|
	\| 3.5857 \| 2.0 \| 188790 \| 3.4442 \| 31.3182 \|
	\| 3.505 \| 3.0 \| 283185 \| 3.3749 \| 29.2214 \|
	\| 3.4688 \| 4.0 \| 377580 \| 3.3439 \| 28.3294 \|
	\| 3.3968 \| 5.0 \| 471975 \| 3.2705 \| 26.3245 \|


	### Framework versions

	- Transformers 4.17.0
	- Pytorch 1.9.1
	- Datasets 2.0.0
	- Tokenizers 0.11.6