BotMed/Llama-3.1-8B-Instruct-BotMed-v2

BotMed-v2 is a fine-tuned version of Llama 3.1 8B Instruct, adapted for medical chatbot use cases using LoRA.

Model Description

BotMed-v2 is a fine-tuned version of Meta's Llama 3.1 8B Instruct, specialized for medical chatbot and question-answering use cases. It was trained using LoRA (Low-Rank Adaptation) on the ruslanmv/ai-medical-chatbot dataset, with the base model loaded in 4-bit NF4 quantization to enable efficient fine-tuning on Kaggle GPUs.

The LoRA adapters (rank 16, alpha 32) target the attention and MLP projection layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj), allowing the model to adapt to medical dialogue patterns while preserving the general instruction-following capabilities of the base Llama 3.1 model. Training was run for up to 3 epochs with early stopping based on evaluation loss, using a cosine learning rate schedule and the paged AdamW 8-bit optimizer.

BotMed-v2 is intended to support use cases such as retrieval-augmented generation (RAG) pipelines for medical knowledge bases, clinical communication training simulations, synthetic medical dialogue generation, and preliminary response drafting in human-in-the-loop healthcare workflows. It is not a certified diagnostic tool and should not be used as a substitute for professional medical advice.

Developed by: Bindupautra Jyotibrat (BJyotibrat)
Shared by: BJyotibrat
Model type: Causal language model, fine-tuned with LoRA
Language(s) (NLP): English
License: Llama 3.1 Community License
Finetuned from model: meta-llama/Meta-Llama-3.1-8B-Instruct

Model Sources

Repository: GitHub Repository
Demo: BotMed Official Website

Uses

Direct Use

Intended for medical chatbot / medical question-answering conversational use cases.

Downstream Use

We envision several potential downstream uses for Llama-3.1-8B-Instruct-BotMed-v2 that extend beyond direct conversational question answering. These potential applications include:

(1) Retrieval-Augmented Generation (RAG) pipelines, serving as a domain-adapted generator integrated with verified medical knowledge bases,

(2) medical education and training, simulating doctor-patient interactions for clinical communication practice,

(3) synthetic data generation, creating synthetic dialogue datasets in conjunction with an evaluator model, and

(4) preliminary query drafting, acting as an assistive tool to generate initial responses for human-in-the-loop healthcare workflows.

(Note: This model is built with Llama and distributed under the Llama 3.1 Community License. Any downstream applications, derivative models, or software integrating this model must comply with Meta's Acceptable Use Policy and include the required "Built with Llama" attribution.)

Out-of-scope uses

We caution potential users of the model that Llama-3.1-8B-Instruct-BotMed-v2 is not a certified medical diagnostic tool, unlike verified clinical software. As such, it is completely unsafe and inappropriate for end-users that require actual medical advice, diagnosis, or treatment, for which a licensed human medical professional (such as a doctor or physician) would be the only appropriate resource.

Additionally, any use of this model must strictly adhere to the Llama 3.1 Acceptable Use Policy. Use cases that violate this policy, or commercial applications exceeding 700 million monthly active users without explicit permission from Meta, are strictly out of scope.

Bias, Risks, and Limitations

Fabricated attribution: Because the training data (ruslanmv/ai-medical-chatbot) consists of real doctor-patient Q&A transcripts, the model has learned to sign off responses with fabricated doctor names, specialties, and closing remarks (e.g. "Regards, Dr S.R.Raveendran, Sexologist"). These are stylistic artifacts inherited from training data, not real credentialed input, and should never be presented to end-users as if they came from an actual licensed physician.
Lexical vs. semantic accuracy gap: Automated evaluation on a 100-question held-out test set shows relatively low lexical overlap with reference answers (F1: 0.1142, ROUGE-1: 0.1834, ROUGE-L: 0.1060) but moderate-to-high semantic similarity (Cosine Similarity: 0.5199, BERTScore F1: 0.7415). This indicates the model tends to phrase answers differently from the reference dataset while still capturing related medical concepts — it should not be assumed that semantically-similar output is clinically equivalent to the reference answer.
No clinical validation: Evaluation metrics used (F1, ROUGE, Cosine Similarity, BERTScore) measure textual/semantic similarity to reference answers, not medical correctness. The model has not been evaluated by medical professionals or validated against clinical guidelines.
Dataset-driven bias: The model's knowledge and tone are shaped entirely by the ruslanmv/ai-medical-chatbot dataset, which may not reflect current medical guidelines, may skew toward specific conditions/specialties represented in that dataset, and may contain its own biases or inaccuracies.
Hallucination risk: As with any LLM, the model can generate plausible-sounding but incorrect or incomplete medical information.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases, and limitations of the model. In particular:

Do not present model-generated doctor names, credentials, or signatures as genuine.
Always pair model output with human review, especially in any workflow that reaches an end-user seeking health guidance.
Treat outputs as a drafting/assistive aid, not a source of medical truth.
Consider domain-specific fact-checking or RAG grounding against verified medical sources before deploying in production.

How to Get Started with the Model

Use the code below to get started with the model.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "BJyotibrat/Llama-3.1-8B-Instruct-BotMed-v2"  # full merged model (base + LoRA already merged)

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
    dtype=torch.float16,
)

model.eval()

prompts = [
    "What are the symptoms of diabetes?",
    "How can high blood pressure be managed?"
]

for idx, t in enumerate(prompts, start=1):
    print(f"\nPrompt {idx}: {t}\n")

    messages = [{"role": "user", "content": t}]
    prompt = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )

    inputs = tokenizer(
        prompt,
        return_tensors="pt",
        truncation=True,
        max_length=2048
    ).to("cuda")

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=512,
            do_sample=True,
            temperature=0.7,
            top_p=0.9,
            repetition_penalty=1.2,
            pad_token_id=tokenizer.eos_token_id
        )

    input_length = inputs["input_ids"].shape[1]
    answer = tokenizer.decode(
        outputs[0][input_length:],
        skip_special_tokens=True,
        clean_up_tokenization_spaces=False
    ).strip()

    print(f"Answer {idx}: {answer}\n")

Training Details

Training Data

Fine-tuned on the ruslanmv/ai-medical-chatbot dataset. The dataset was split into train/test sets using a 95/5 split (test_size=0.05, seed=65).

Training Procedure

Fine-tuned using LoRA (Low-Rank Adaptation) on Kaggle GPUs, with the base model loaded in 4-bit quantization via BitsAndBytes.

Preprocessing

Quantization (BitsAndBytes):

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    base_model,
    quantization_config=bnb_config,
    device_map="auto",
    attn_implementation=attn_implementation,
    dtype=torch.float16,
)

model = prepare_model_for_kbit_training(model)

LoRA Configuration:

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
)

model = get_peft_model(model, peft_config)
model.print_trainable_parameters()

Training Hyperparameters

Training regime: fp16=False, bf16=False (mixed-precision compute via bnb_4bit_compute_dtype=torch.float16)
Batch size: 2 per device (train and eval), with gradient accumulation steps of 4
Optimizer: paged_adamw_8bit
Epochs: up to 3, with early stopping (added on top of load_best_model_at_end=True, metric_for_best_model="eval_loss") — training was manually stopped after the 2nd epoch showed no further improvement
Evaluation/Save strategy: every 200 steps
Logging: every 50 steps
Warmup steps: 36
Learning rate: 1e-4, cosine scheduler
Gradient checkpointing: enabled
Checkpoints kept: 3 (save_total_limit=3)
Experiment tracking: Weights & Biases (wandb)

training_arguments = SFTConfig(
    output_dir=new_model,

    per_device_train_batch_size=2,
    per_device_eval_batch_size=2,
    gradient_accumulation_steps=4,

    optim="paged_adamw_8bit",
    num_train_epochs=3,              # 3 epochs max, early stopping will protect

    eval_strategy="steps",
    eval_steps=200,
    save_strategy="steps",
    save_steps=200,

    logging_steps=50,

    warmup_steps=36,
    learning_rate=1e-4,
    lr_scheduler_type="cosine",

    fp16=False,
    bf16=False,

    report_to="wandb",
    gradient_checkpointing=True,

    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,

    save_total_limit=3,              # keep one checkpoint per epoch
)

Speeds, Sizes, Times

Final Weights & Biases run summary (best checkpoint, selected on lowest eval_loss):

Metric	Value
eval/loss	2.01318
eval/entropy	1.92125
eval/mean_token_accuracy	0.56194
eval/num_tokens	3,259,218
eval/runtime (s)	302.3117
eval/samples_per_second	1.654
eval/steps_per_second	0.827
train/entropy	1.87502
train/epoch	1.34695
total_flos	1.7632157585149133e+17

Full step-by-step run history is available as a CSV export: wandb_run_history - BotMed v2.csv.

Evaluation

Testing Data, Factors & Metrics

Testing Data

A held-out set of 100 medical questions, with model-generated answers compared against reference answers. The full question set, generated answers, and per-question scoring are available here: BotMed_v2_QA_Results.csv.

Factors

Evaluation was conducted on general open-domain medical Q&A prompts; no disaggregation by medical specialty, demographic, or question type was performed.

Metrics

F1 Score — token-level overlap between generated and reference answers.
ROUGE-1 / ROUGE-L — unigram and longest-common-subsequence overlap, measuring lexical similarity to reference answers.
Cosine Similarity — embedding-based similarity between generated and reference answers.
BERTScore (Precision / Recall / F1) — contextual embedding similarity (via distilbert-base-uncased), capturing semantic alignment even when wording differs.

Results

Metric	Score
F1 Score	0.1142
ROUGE-1	0.1834
ROUGE-L	0.1060
Cosine Similarity	0.5199
BERTScore Precision	0.7588
BERTScore Recall	0.7272
BERTScore F1	0.7415

Full per-question predictions and scores (100 questions) are available in BotMed_v2_QA_Results.csv.

Summary

Lexical overlap metrics (F1, ROUGE) are low, indicating the model rarely reproduces the reference answers' exact wording. However, BERTScore and Cosine Similarity indicate moderate-to-strong semantic alignment, suggesting the model generally captures the intent and relevant medical concepts of the reference answers while phrasing them differently. This is consistent with the free-form, conversational nature of the training data.

Example Outputs

Prompt 1: What are the symptoms of diabetes?

Answer 1: Symptoms include increased thirst and urination, blurring or double vision, fatigue, slow healing cuts/sores, etc. Also, if you have a family history of diabetes then it would be good to get yourself tested as well for early detection and treatment. You can also visit your doctor with these details so that he/she may advise accordingly.

Prompt 2: How can high blood pressure be managed?

Answer 2: High BP needs to be controlled by medications such as diuretics, calcium channel blockers like amlodipine or ACE inhibitors like enalapril. Lifestyle modification includes dietary restrictions, avoidance of salt in diet, regular exercise, etc.

(Note: raw model outputs include fabricated sign-offs such as doctor names and specialties)

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Hardware Type: Kaggle GPUs
Hours used: 8 hours
Cloud Provider: Kaggle

Technical Specifications

Model Architecture and Objective

Llama 3.1 8B Instruct architecture, fine-tuned via LoRA (rank 16, alpha 32) targeting the q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj modules, for medical chatbot dialogue generation. Base model loaded in 4-bit NF4 quantization for training.