SyedaArisha/medical-llama3-lora

Medical LLaMA 3 — Fine-tuned with QLoRA + Unsloth

A Llama 3 8B model fine-tuned on a medical Q&A dataset for accurate medical question answering.

Model Details

• Base Model: meta-llama/Meta-Llama-3-8B
• Fine-tuning Method: QLoRA (4-bit quantization + LoRA adapters)
• Training Framework: Unsloth
• Dataset: medalpaca/medical_meadow_medical_flashcards
• Training Examples: 8,000 medical Q&A pairs
• Trainable Parameters: 20,971,520 of 8,051,232,768 (0.26%)
• Training Platform: Google Colab T4 GPU

Performance

Training loss reduced from 1.154 → 0.763 across 1,000 steps.

Example Output

Question: What are the early symptoms of diabetes?

Answer: Polyuria (increased urination) and polydipsia (increased thirst) are the early symptoms of diabetes. These symptoms are caused by the body's attempt to eliminate excess glucose from the bloodstream...

Usage

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "SyedaArisha/medical-llama3-lora",
    max_seq_length = 1024,
    load_in_4bit = True,
)

FastLanguageModel.for_inference(model)

inputs = tokenizer(
    """Below is a medical question. Answer it accurately.

### Question:
What are the symptoms of hypertension?

### Answer:
""",
    return_tensors = "pt"
).to("cuda")

outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Fine-tuned using QLoRA with the following configuration:

• LoRA rank: 8
• LoRA alpha: 8
• Quantization: 4-bit
• Epochs: 1
• Batch size: 2 (effective: 8 with gradient accumulation)
• Learning rate: 2e-4

Developed By

Syeda Arisha Hassan — Software Engineering Student, COMSATS University Islamabad