ClinicalQwen-1.5B-Medical

A QLoRA fine-tuned version of Qwen2.5-1.5B-Instruct for medical question answering.

Disclaimer: This model is for educational and research purposes only. Do not use it as a substitute for professional medical advice, diagnosis, or treatment. Always consult a qualified healthcare professional.

How to Use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

# Load base model with 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-1.5B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)

# Load fine-tuned adapter
model = PeftModel.from_pretrained(base_model, "Coddieharsh/ClinicalQwen-1.5B-Medical")
tokenizer = AutoTokenizer.from_pretrained("Coddieharsh/ClinicalQwen-1.5B-Medical")

# Generate response
def ask(question):
    prompt = (
        "<|im_start|>system\n"
        "You are a knowledgeable medical assistant. Provide accurate, evidence-based "
        "medical information. Always recommend consulting a healthcare professional "
        "for personal medical advice.<|im_end|>\n"
        f"<|im_start|>user\n{question}<|im_end|>\n"
        "<|im_start|>assistant\n"
    )
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=512,
            temperature=0.7,
            do_sample=True,
            top_p=0.9,
            repetition_penalty=1.1,
            pad_token_id=tokenizer.eos_token_id,
        )
    response = tokenizer.decode(outputs[0], skip_special_tokens=False)
    return response.split("<|im_start|>assistant")[-1].split("<|im_end|>")[0].strip()

print(ask("What are the symptoms of Type 2 diabetes?"))

Training Details

Parameter	Value
Base Model	Qwen/Qwen2.5-1.5B-Instruct
Fine-tuning Method	QLoRA (4-bit NF4)
Dataset	lavita/medical-qa-datasets
Training Samples	5,000
Epochs	1
Final Loss	1.8991
Training Time	~2.4 hours
Hardware	Google Colab T4 (16GB)
LoRA Rank (r)	8
LoRA Alpha	16
Learning Rate	2e-4
Batch Size (effective)	16 (2 × 8 grad accum)
Optimizer	paged_adamw_8bit
Scheduler	Cosine
Max Sequence Length	512
Trainable Parameters	9.2M / 1,552M (< 1%)

Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Evaluation Results

Evaluated on 10 medical benchmark questions:

Metric	Score	Note
BERTScore F1	0.5913	Semantic similarity (primary metric)
Perplexity	3.14	Fluency — lower is better
Keyword Coverage	86.0%	Medical term coverage
ROUGE-L	0.1219	Surface overlap (low = paraphrasing, expected)
BLEU	0.0099	N-gram precision (low = paraphrasing, expected)

Low BLEU/ROUGE is expected for generative medical assistants — the model paraphrases rather than copying reference text verbatim.

Sample Output

Q: What are the symptoms of Type 2 diabetes?

A: Type 2 diabetes is characterized by high blood sugar levels over time. Common symptoms include:

• Frequent urination (polyuria)
• Increased thirst (polydipsia)
• Fatigue and low energy
• Blurred vision
• Slow-healing wounds
• Numbness or tingling in hands and feet

If you experience these symptoms, consult a healthcare professional for proper diagnosis.

Limitations

• Trained on 5,000 samples (1 epoch) — a larger dataset and more epochs would improve accuracy
• Not validated by medical professionals — outputs may contain inaccuracies
• Should not be used for clinical decision-making
• Best suited for general health information queries

Repository

Full training code, notebooks, and evaluation scripts: github.com/HarshTomar1234/ClinicalQwen