anphiriel/peer-model-0.04

Fine-Tuned Llama-3.1-8B-Instruct with QLoRA

This model was fine-tuned on anphiriel/peer-dataset-0.04 using QLoRA with 4-bit quantization.

• Base Model: meta-llama/Llama-3.1-8B-Instruct
• Training Data Size: 16256 examples # Use actual dataset size
• Training Epochs: 10 # Use value from args
• Learning Rate: 0.0002 # Use value from args
• Marker: <|eot_id|>
• LoRA Config: r=16, alpha=32, dropout=0.05
• Quantization: 4-bit NF4, bfloat16 compute
• Effective Batch Size: 16
• Evaluation Metric for Best Model: eval_loss
• Early Stopping Patience: 3

This model is intended for use as a chatbot/sales assistant. It was trained on data formatted using the Llama-3.1 chat template.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "anphiriel/peer-0.04-merged"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
  model_id,
  torch_dtype=torch.bfloat16, # Use bfloat16 for consistency and A100
  device_map="auto"
)

# Correct Llama-3.1 chat template format
messages = [
  {"role": "user", "content": "What is file distribution?"},
  # Add previous turns if applicable
]

input_ids = tokenizer.apply_chat_template(
  messages,
  add_generation_prompt=True,
  return_tensors="pt"
).to(model.device)

terminators = [
  tokenizer.eos_token_id,
  tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

outputs = model.generate(
  input_ids,
  max_new_tokens=256, # Adjust as needed
  eos_token_id=terminators,
  do_sample=True,
  temperature=0.0, # Use 0.0 for deterministic/factual, higher for creative
  top_p=0.9,
)

response = outputs[0][input_ids.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))