germeo-7b-awq / README.md
aari1995's picture
Update README.md
dda3460 verified
metadata
language:
  - de
pipeline_tag: text-generation
tags:
  - awq
  - autoawq
license: apache-2.0

WIP

(Please bear with me, this model will get better and get a license soon)

Hermes + Leo + German AWQ = Germeo

Germeo-7B-AWQ

A German-English understanding, but German-only speaking model merged from Hermeo-7B.

Model details

Quantization Procedure and Use Case:

The speciality of this model is that it solely replies in German, independently from the system message or prompt. Within the AWQ-process I introduced OpenSchnabeltier as calibration data for the model to stress the importance of German Tokens.

Usage

Setup in autoawq

# setup [autoawq](https://github.com/casper-hansen/AutoAWQ)
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer, TextStreamer

quant_path = "aari1995/germeo-7b-awq"

# Load model
model = AutoAWQForCausalLM.from_quantized(quant_path, fuse_layers=True)
tokenizer = AutoTokenizer.from_pretrained(quant_path, trust_remote_code=True)

Setup in transformers (works in colab)

# pip install [autoawq](https://github.com/casper-hansen/AutoAWQ) and pip install --upgrade transformers
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

quant_path = "aari1995/germeo-7b-awq"

# Load model
model = AutoModelForCausalLM.from_pretrained(quant_path, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(quant_path, trust_remote_code=True)

Inference:

streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

# Convert prompt to tokens
prompt_template = """<|im_start|>system
Du bist ein hilfreicher Assistent.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant"""

prompt = "Schreibe eine Stellenanzeige für Data Scientist bei AXA!"

tokens = tokenizer(
    prompt_template.format(prompt=prompt), 
    return_tensors='pt'
).input_ids.cuda()

# Generate output
generation_output = model.generate(
    tokens, 
    streamer=streamer,
    max_new_tokens=1012
)
# tokenizer.decode(generation_output.flatten())

FAQ

The model continues after the reply with user inputs:

To solve this, you need to implement a custom stopping criteria:

from transformers import StoppingCriteria
class GermeoStoppingCriteria(StoppingCriteria):
  def __init__(self, target_sequence, prompt):
      self.target_sequence = target_sequence
      self.prompt=prompt

  def __call__(self, input_ids, scores, **kwargs):
      # Get the generated text as a string
      generated_text = tokenizer.decode(input_ids[0])
      generated_text = generated_text.replace(self.prompt,'')
      # Check if the target sequence appears in the generated text
      if self.target_sequence in generated_text:
          return True  # Stop generation

      return False  # Continue generation

  def __len__(self):
      return 1

  def __iter__(self):
      yield self

This then expects your input prompt (formatted as given into the model), and a stopping criteria, in this case the im_end token. Simply add it to the generation:

generation_output = model.generate(
    tokens, 
    streamer=streamer,
    max_new_tokens=1012,
    stopping_criteria=GermeoStoppingCriteria("<|im_end|>", prompt_template.format(prompt=prompt))
)

Acknowledgements and Special Thanks

Evaluation and Benchmarks (German only)

German benchmarks

German tasks: MMLU-DE Hellaswag-DE ARC-DE Average
Models / Few-shots: (5 shots) (10 shots) (24 shots)
7B parameters
llama-2-7b 0.400 0.513 0.381 0.431
leo-hessianai-7b 0.400 0.609 0.429 0.479
bloom-6b4-clp-german 0.274 0.550 0.351 0.392
mistral-7b 0.524 0.588 0.473 0.528
leo-mistral-hessianai-7b 0.481 0.663 0.485 0.543
leo-mistral-hessianai-7b-chat 0.458 0.617 0.465 0.513
DPOpenHermes-7B-v2 0.517 0.603 0.515 0.545
hermeo-7b 0.511 0.668 0.528 0.569
germeo-7b-awq (this model) 0.522 0.651 0.514 0.563
13B parameters
llama-2-13b 0.469 0.581 0.468 0.506
leo-hessianai-13b 0.486 0.658 0.509 0.551
70B parameters
llama-2-70b 0.597 0.674 0.561 0.611
leo-hessianai-70b 0.653 0.721 0.600 0.658

German reply rate benchmark

The fraction of German reply rates according to this benchmark

Models: German Response Rate
hermeo-7b tba
germeo-7b-awq (this model) tba

Additional Benchmarks:

TruthfulQA-DE: 0.508