|
--- |
|
license: cc-by-nc-sa-4.0 |
|
datasets: |
|
- SebastianBodza/Ger_WizardLM_evol_instruct_70k_V0 |
|
language: |
|
- de |
|
--- |
|
# DElefant: |
|
<img src="https://huggingface.co/SebastianBodza/DElefant/resolve/main/badge_gerlefant.png" style="max-width:200px"> |
|
DElefant is a LLM developed for instruction tuned German interactions. This version is built on top of the adapted BLOOM version from <a href="https://huggingface.co/malteos/bloom-6b4-clp-german">Malte Ostendorff</a> with a opus-mt translated and afterwards filtered <a href="https://huggingface.co/datasets/SebastianBodza/Ger_WizardLM_evol_instruct_70k_V0">WizardLM</a> dataset. The evolved dataset led to SOTA english LLMs and we hope by incoperating the dataset to a german base model we can leverage the capabilities for various tasks including Code generation. |
|
Due to limitation in translation, the comments inside of the code blocks remained english, however the Coding was kept in working condition. |
|
|
|
## Model Description: |
|
Full-Finetuning of the German-BLOOM model on an RTX 3090 with the translated WizardLM Dataset. |
|
|
|
## Roadmap: |
|
If there is sufficient demand, additional adjustments can be made: |
|
- Native German generated dataset |
|
- Full Fine-Tuning of larger LLMs e.g. Falcon, Starcoderplus, ... |
|
|
|
## How to use: |
|
Prompt-Template: |
|
``` |
|
{instruction}\n\n### Response: |
|
``` |
|
Code example for inference: |
|
``` |
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
tokenizer = AutoTokenizer.from_pretrained("SebastianBodza/DElefant") |
|
model = AutoModelForCausalLM.from_pretrained("SebastianBodza/DElefant", device_map="auto") |
|
frage = "Wie heißt der Bundeskanzler?" |
|
prompt = f"{frage}\n\n### Response:" |
|
|
|
txt = tokenizer(prompt, return_tensors="pt").to("cuda") |
|
txt = model.generate(**txt, |
|
max_new_tokens=256, |
|
eos_token_id=tokenizer.eos_token_id) |
|
tokenizer.decode(txt[0], skip_special_tokens=True) |
|
``` |
|
## Training: |
|
Training was based on Llama-X with the adaptions of WizardLMs training script. |
|
``` |
|
deepspeed Llama-X/src/train_freeform.py \ |
|
--model_name_or_path malteos/bloom-6b4-clp-german \ |
|
--data_path ger_alpaca_evol_instruct_70k_e.json \ |
|
--output_dir ./full_finetune \ |
|
--num_train_epochs 2 \ |
|
--model_max_length 2048 \ |
|
--per_device_train_batch_size 2 \ |
|
--per_device_eval_batch_size 1 \ |
|
--gradient_accumulation_steps 8 \ |
|
--evaluation_strategy "no" \ |
|
--save_strategy "steps" \ |
|
--save_steps 400 \ |
|
--save_total_limit 3 \ |
|
--learning_rate 2e-5 \ |
|
--warmup_steps 2 \ |
|
--logging_steps 2 \ |
|
--lr_scheduler_type "cosine" \ |
|
--report_to "tensorboard" \ |
|
--gradient_checkpointing True \ |
|
--deepspeed deepspeed.json \ |
|
--bf16 True |
|
``` |
|
<img src="https://huggingface.co/SebastianBodza/DElefant/resolve/main/train_loss_DElefant.svg" style="max-width:350px"> |
|
|
|
|
