metadata

base_model: meta-llama/Llama-3.1-8B-Instruct
library_name: transformers
model_name: Llama-3.1-8B-KAM
tags:
  - trl
  - Llama
  - sft
  - generated_from_trainer
licence: license

Model Card for Llama-3.1-8B-KAM

This model is a fine-tuned version of meta-llama/Llama-3.1-8B-Instruct on the None dataset.

Model description

More information needed

Quick start

from transformers import pipeline

question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
generator = pipeline("text-generation", model="MaRyAm1295/Llama-3.1-8B-KAM", device="cuda")
output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
print(output["generated_text"])

Training procedure

This model was trained with SFT.

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0002
train_batch_size: 1
eval_batch_size: 8
seed: 3407
gradient_accumulation_steps: 16
total_train_batch_size: 8
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 20
training_steps: 500
mixed_precision_training: Native AMP

Training results

Step Training Loss

50 2.158200
100 1.845900
150 1.832200
200 1.805300
250 1.783800
300 1.767500
350 1.744800
400 1.745600
450 1.749500
500 1.756100

Framework versions

TRL: 0.12.0
Transformers: 4.46.2
Pytorch: 2.4.0
Datasets: 3.0.1
Tokenizers: 0.20.0