|
--- |
|
license: mit |
|
datasets: |
|
- wikimedia/wikipedia |
|
language: |
|
- ka |
|
- en |
|
pipeline_tag: text-generation |
|
--- |
|
|
|
# GPT-2Geo: Georgian Language Model ๐ฌ๐ช |
|
|
|
> โ ๏ธ This GPT-2Geo model is not fully trained due to hardware constraints. It has been trained on a subset of 1000 samples for 20 epochs. The model's capabilities and performance are indicative within these limitations. Future iterations may benefit from extended training on more extensive datasets. Please be mindful of these training constraints when utilizing the model. |
|
|
|
## Overview |
|
|
|
GPT-2Geo is a powerful language model tailored for the Georgian language, built upon OpenAI's GPT-2 architecture. This model is designed for various natural language processing tasks, including text generation and understanding. [Github (training script)](https://github.com/Kuduxaaa/gpt2-geo) |
|
|
|
## Features |
|
|
|
- **Georgian Language Model:** Specifically trained to understand and generate text in the Georgian language. |
|
- **GPT-2 Architecture:** Built upon OpenAI's GPT-2, providing a versatile and efficient language model. |
|
- **Easy Integration:** Seamless integration with the Hugging Face Transformers library. |
|
|
|
## Training Information |
|
|
|
### Environment: |
|
- **GPU:** Nvidia T4 (15GB) |
|
- **Model Memory Requirement:** Minimum 13.5GB |
|
|
|
### Training Configuration: |
|
- **Number of Epochs:** 20 |
|
- **Time Consumed:** 49 minutes |
|
|
|
 |
|
|
|
### Training Progress: |
|
The GPT-2Geo model underwent training in a high-performance environment utilizing the **Nvidia T4 GPU** with **15GB** of dedicated memory. This powerful hardware met the minimum model memory requirement of **13.5GB**, ensuring optimal performance during the training process. |
|
|
|
The training configuration included **20 epochs**, allowing the model to iteratively learn from the dataset. The entire training procedure was completed in a time-efficient manner, consuming approximately **49 minutes**. |
|
|
|
For detailed insights into the model's performance, refer to the training logs, which capture key metrics such as validation loss over epochs. This information provides users with a comprehensive understanding of the training environment, configuration, and progress. |
|
|
|
Ensure that your GPU environment is correctly configured to harness the full potential of the available hardware during the training phase. **Before start training process it needs to preprocess text data and it will added in future** |
|
|
|
## Example Usage |
|
|
|
```python |
|
import torch |
|
|
|
from transformers import GPT2LMHeadModel, GPT2Config, ElectraTokenizerFast |
|
|
|
model_name = 'Kuduxaaa/gpt2-geo' |
|
model = GPT2LMHeadModel.from_pretrained(model_name) |
|
tokenizer = ElectraTokenizerFast.from_pretrained(model_name) |
|
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') |
|
model.to(device) |
|
|
|
prompt = 'แฅแแ แแฃแ แแแแแแแแแแจแ ' |
|
|
|
input_ids = tokenizer.encode(prompt, return_tensors='pt').to(device) |
|
output = model.generate( |
|
input_ids, |
|
max_length = 100, |
|
num_beams = 5, |
|
no_repeat_ngram_size = 2, |
|
top_k = 50, |
|
top_p = 0.95, |
|
temperature = 0.7 |
|
) |
|
|
|
result = tokenizer.decode(output[0], skip_special_tokens=True) |
|
print(result) |
|
# แฅแแ แแฃแ แแแแแแแแแแจแ, แแแแแแแก แแแ แกแแแแแแแ แแ. แแแแแแ แแแแแแจแแ แแแฃแแ แแแแฃแ แ แฌแแ แแแจแแแแก, แ แแแแแแช แฌแแ แแแแจแแ แแแแ แแแ แแแแช, แ แแ แแ แแแ แแแแจแ แแ แกแฎแแ แกแฎแแ. แแแ แแแแ แแแแแแแ แฌแแ แแแแแแแแก แแแแแแแแ แแ แแแ, แ แแแแ แช แกแแจแฃแแแแแแแ, แกแแคแฃแซแแแแ แฌแแ แแแแแแก แฌแแ แกแฃแแจแ. แแแขแแ แแขแฃแ แ แฌแแ แแแแแแแแแแแก แแแ |
|
|
|
``` |
|
|
|
## Acknowledgments |
|
|
|
This project is made possible by the contributions of Nika Kudukashvili and the open-source community. Special thanks to OpenAI for the GPT-2 architecture and `jnz/electra-ka` for georgian tokenizer. |
