Hugging Face's logo

caretech-owl
/
leo-hessionai-7B-quotes-gguf

Text2Text Generation
GGUF
German
Model card Files Community
leo-hessionai-7B-quotes-gguf / README.md
costrau's picture
costrau
Update README.md
d9519a5
preview code
|
raw
history blame contribute delete
No virus
3.05 kB
metadata
license: llama2
base_model: LeoLM/leo-hessianai-7b
datasets:
  - caretech-owl/wikiquote-de-quotes
language:
  - de
pipeline_tag: text2text-generation

Model Card for Model ID

This model is trained to generate german quotes for a given author. The full model can be tested at spaces/caretech-owl/quote-generator-de, here we provide a full model with a 8 bit quantization.

Model Details

Model Description

This fine-tuned model has been trained on the caretech-owl/wikiquote-de-quotes dataset. The model was trained on a prompt like this


prompt_format = "<|im_start|>system\
Dies ist eine Unterhaltung zwischen einem\
intelligenten, hilfsbereitem KI-Assistenten und einem Nutzer.
Der Assistent gibt Antworten in Form von Zitaten.<|im_end|>\n\
<|im_start|>user\
Zitiere {author}<|im_end|>\n<\
|im_start|>assistant\n{quote}<|im_end|>\n"

Where author is itended to be provided by the user, the quote is of format quote + " - " + author. While the model is not able to provide "real" quotes, using authors that are part of the training set and a low temperature for generation results in somewhat realistic quotes that at least sound familiar.

Uses 

import torch
from ctransformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained(
    "caretech-owl/leo-hessionai-7B-quotes-gguf", model_type="llama")

system_prompt = """Dies ist eine Unterhaltung zwischen \
einem intelligenten, hilfsbereitem \
KI-Assistenten und einem Nutzer.
Der Assistent gibt Antworten in Form von Zitaten."""
prompt_format = "<|im_start|>system\n{system_prompt}\
<|im_end|>\n<|im_start|>user\nZitiere {prompt}\
<|im_end|>\n<|im_start|>assistant\n"

def get_quote(author:str, max_new_tokens:int=200):
    query = prompt_format.format(system_prompt=system_prompt, prompt= author)
    output = base_model(query, stop='<|im_end|>', max_new_tokens=max_new_tokens, temperature=0.2, top_k=10)
    print(output)

get_quote("Heinrich Heine")

Training procedure

The following bitsandbytes quantization config was used during training:

  • quant_method: gptq
  • bits: 8
  • tokenizer: None
  • dataset: None
  • group_size: 32
  • damp_percent: 0.1
  • desc_act: True
  • sym: True
  • true_sequential: True
  • use_cuda_fp16: False
  • model_seqlen: None
  • block_name_to_quantize: None
  • module_name_preceding_first_block: None
  • batch_size: 1
  • pad_token_id: None
  • use_exllama: True
  • max_input_length: None
  • exllama_config: {'version': <ExllamaVersion.ONE: 1>}
  • cache_block_outputs: True

Framework versions

  • PEFT 0.6.2