Yale-BIDS-Chen/medpmc-clip-l-14_jun24_v1

MedPMC-CLIP

MedPMC-CLIP is a medical vision-language model based on the OpenCLIP ViT-L-14 architecture. The model was presented in the paper MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models.

The model was trained on the MedPMC-11M dataset, a carefully curated collection of approximately 11 million image-caption pairs derived from biomedical literature. Across a wide range of evaluations, MedPMC-CLIP consistently outperforms existing baseline models, including zero-shot medical image classification on 26 public benchmarks and zero-shot image retrieval on an internal clinical dermatology dataset.

For additional details on model training and benchmark results, please refer to the paper and GitHub repository.

To reproduce the evaluation results reported in the paper, please refer to medpmc-clip-eval.

This repository provides the checkpoint in OpenCLIP format. Text inputs should be tokenized using the default OpenCLIP tokenizer for ViT-L-14.

tokenizer = open_clip.get_tokenizer("ViT-L-14")

Files

• open_clip_model.safetensors: OpenCLIP-format model checkpoint
• inference_example.py: example code for image-text similarity
• requirements.txt: minimal dependencies

Usage

import torch
import open_clip
from huggingface_hub import hf_hub_download
from PIL import Image

model_name = "ViT-L-14"
device = "cuda" if torch.cuda.is_available() else "cpu"

repo_id = "Yale-BIDS-Chen/medpmc-clip-l-14_jun24_v1"

ckpt_path = hf_hub_download(
    repo_id=repo_id,
    filename="open_clip_model.safetensors",
)

model, _, preprocess = open_clip.create_model_and_transforms(
    model_name,
    pretrained=ckpt_path,
    device=device,
)

model.eval()
tokenizer = open_clip.get_tokenizer(model_name)

image = preprocess(
    Image.open("/path/to/example.jpg").convert("RGB")
).unsqueeze(0).to(device)

text = tokenizer([
    "fundus photograph",
    "chest radiograph",
    "histopathology image",
]).to(device)

with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)

    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)

    similarity = image_features @ text_features.T

print(similarity)

License

The model is released for non-commercial research use under CC BY-NC-SA 4.0.

Citation

@article{kim2026medpmc,
  title={MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models},
  author={Kim, Hyunjae and Kim, Dain and Xiao, Pan and Applebaum, Serina S and Chung, Younjoon and Ai, Xuguang and Yin, Yu and Jiang, Roy and Du, Yuexi and Wei, Yawen and others},
  journal={arXiv preprint arXiv:2607.07673},
  year={2026}
}

Questions?

For questions or feedback, please contact Hyunjae Kim at hyunjae.kim@yale.edu.