🏥 MedGPT — Medical Visual Question Answering

AI-powered medical image analysis with visual explanations

Features • Architecture • Installation • Usage • Results • Datasets

---

✨ Features

• Vision-Language AI — Powered by Qwen3-VL-8B fine-tuned on 150K+ medical VQA samples
• Grad-CAM Heatmaps — Visual explanations showing where the model focuses
• Multi-Modality — X-ray, CT, MRI, Ultrasound, and Pathology slides
• React + Vite Frontend — Modern, responsive web interface with glassmorphism design
• FastAPI Backend — GPU-accelerated inference with REST API
• Dashboard — Interactive training curves, accuracy charts, and metrics visualizations

🏗️ Architecture

MedGPT/
├── frontend/              # React + Vite frontend
│   ├── src/
│   │   ├── pages/         # Home, Analyze, Dashboard, About
│   │   ├── components/    # Navbar, Footer
│   │   └── styles/        # Global CSS design system
│   └── dist/              # Production build
├── backend/               # FastAPI server
│   └── server.py          # API endpoints + React SPA serving
├── models/                # Model definitions
│   ├── medgpt.py          # MedGPT model class
│   └── explainability.py  # Grad-CAM implementation
├── training/              # Training pipeline
│   ├── train.py           # Pre-training + fine-tuning script
│   ├── evaluate.py        # Evaluation metrics
│   └── visualize.py       # Generate training curves & charts
├── inference/             # Inference utilities
│   └── predict.py         # CLI prediction tool
├── data/                  # Datasets
│   ├── prepare_data.py    # Download & preprocess datasets
│   ├── dataset.py         # PyTorch dataset classes
│   └── processed/         # Processed JSON splits
├── configs/
│   └── config.yaml        # All configuration settings
├── checkpoints/           # Trained model checkpoints
│   └── finetune/
│       └── best_model/    # Best LoRA adapter
└── results/               # Evaluation results & visualizations
    └── figures/           # Training curves, charts, heatmaps

📊 Results

Metric	Score
Overall Accuracy	78.5%
Yes/No Accuracy	87.1%
Open-ended Accuracy	73.9%
BLEU-1	82.8%
ROUGE-L	80.6%
Token F1	81.2%

Training Details

Parameter	Value
Base Model	Qwen3-VL-8B-Instruct
Fine-tuning Method	LoRA (rank=64, alpha=128)
Trainable Parameters	210M / 9B (2.3%)
Precision	bfloat16
Training Epochs	3
Hardware	NVIDIA H200 (141GB VRAM)
Training Time	~5.6 hours

📦 Datasets

Dataset	Samples	Stage
PMC-VQA	~140K	Pre-training
VQA-RAD	~3.5K	Fine-tuning
SLAKE	~14K	Fine-tuning
PathVQA	~32K	Fine-tuning

🚀 Installation

Prerequisites

• Python 3.10+
• NVIDIA GPU with 24GB+ VRAM (48GB+ recommended)
• Node.js 18+ (for frontend)

Setup

# Clone from HuggingFace
git lfs install
git clone https://huggingface.co/bhargavvz/MedGPT
cd MedGPT

# Install Python dependencies
pip install -r requirements.txt

# Download datasets
python data/prepare_data.py --datasets all --output_dir data/processed --validate

# Build frontend
cd frontend && npm install && npm run build && cd ..

💻 Usage

Web Application

# Start the server (API + React frontend on port 8000)
python backend/server.py

# Open: http://localhost:8000

CLI Inference

python inference/predict.py \
    --image path/to/xray.jpg \
    --question "What abnormality is visible?" \
    --adapter_path checkpoints/finetune/best_model

Training

# Full pipeline (pre-training + fine-tuning)
python training/train.py --stage all

# Fine-tuning only
python training/train.py --stage finetune

Evaluation

python training/evaluate.py \
    --adapter_path checkpoints/finetune/best_model \
    --test_file data/processed/finetune_test.json \
    --output_file results/eval_results.json

Generate Visualizations

python training/visualize.py --output_dir results/figures

🐳 Docker

docker compose up -d
# Access at http://localhost:8000

🔗 API Endpoints

Method	Endpoint	Description
GET	/api/health	Health check & model status
POST	/api/predict	Image + question → answer + heatmap
POST	/api/load	Load/reload model
GET	/api/metrics	Evaluation results
GET	/api/training-history	Training loss curves

📄 Tech Stack

• Model: Qwen3-VL-8B + LoRA
• Backend: FastAPI + Uvicorn
• Frontend: React + Vite + Framer Motion + Recharts
• Training: PyTorch + HuggingFace Transformers
• Explainability: Grad-CAM
• Deployment: Docker + NVIDIA CUDA

⚠️ Disclaimer

MedGPT is a research and educational tool. It is NOT intended for clinical diagnosis or medical decision-making. Always consult qualified healthcare professionals for medical advice.

📝 License

This project is for educational and research purposes only.

---

Built with ❤️ by Bhargav