Package: blip-ci
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and GenerationAnnouncement: BLIP is now officially integrated intoLAVIS- a one-stop library for language-and-vision research and applications!This is the PyTorch code of theBLIP paper[blog]. The code has been tested on PyTorch 1.10.
To install the dependencies, runpip install -r requirements.txtCatalog:Inference demoPre-trained and finetuned checkpointsFinetuning code for Image-Text Retrieval, Image Captioning, VQA, and NLVR2Pre-training codeZero-shot video-text retrievalDownload of bootstrapped pre-training datasetsInference demo:Run our interactive demo usingColab notebook(no GPU needed).
The demo includes code for:Image captioningOpen-ended visual question answeringMultimodal / unimodal feature extractionImage-text matchingTry out theWeb demo, integrated intoHuggingface Spaces 🤗usingGradio.Replicate web demo and Docker image is also available atPre-trained checkpoints:Num. pre-train imagesBLIP w/ ViT-BBLIP w/ ViT-B and CapFilt-LBLIP w/ ViT-L14MDownload--129MDownloadDownloadDownloadFinetuned checkpoints:TaskBLIP w/ ViT-BBLIP w/ ViT-B and CapFilt-LBLIP w/ ViT-LImage-Text Retrieval (COCO)Download-DownloadImage-Text Retrieval (Flickr30k)Download-DownloadImage Captioning (COCO)-DownloadDownloadVQADownloadDownload-NLVR2Download--Image-Text Retrieval:Download COCO and Flickr30k datasets from the original websites, and set 'image_root' in configs/retrieval_{dataset}.yaml accordingly.To evaluate the finetuned BLIP model on COCO, run:python -m torch.distributed.run --nproc_per_node=8 train_retrieval.py \
--config ./configs/retrieval_coco.yaml \
--output_dir output/retrieval_coco \
--evaluateTo finetune the pre-trained checkpoint using 8 A100 GPUs, first set 'pretrained' in configs/retrieval_coco.yaml as "https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base.pth". Then run:python -m torch.distributed.run --nproc_per_node=8 train_retrieval.py \
--config ./configs/retrieval_coco.yaml \
--output_dir output/retrieval_cocoImage-Text Captioning:Download COCO and NoCaps datasets from the original websites, and set 'image_root' in configs/caption_coco.yaml and configs/nocaps.yaml accordingly.To evaluate the finetuned BLIP model on COCO, run:python -m torch.distributed.run --nproc_per_node=8 train_caption.py --evaluateTo evaluate the finetuned BLIP model on NoCaps, generate results with: (evaluation needs to be performed on official server)python -m torch.distributed.run --nproc_per_node=8 eval_nocaps.pyTo finetune the pre-trained checkpoint using 8 A100 GPUs, first set 'pretrained' in configs/caption_coco.yaml as "https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base_capfilt_large.pth". Then run:python -m torch.distributed.run --nproc_per_node=8 train_caption.pyVQA:Download VQA v2 dataset and Visual Genome dataset from the original websites, and set 'vqa_root' and 'vg_root' in configs/vqa.yaml.To evaluate the finetuned BLIP model, generate results with: (evaluation needs to be performed on official server)python -m torch.distributed.run --nproc_per_node=8 train_vqa.py --evaluateTo finetune the pre-trained checkpoint using 16 A100 GPUs, first set 'pretrained' in configs/vqa.yaml as "https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base_capfilt_large.pth". Then run:python -m torch.distributed.run --nproc_per_node=16 train_vqa.pyNLVR2:Download NLVR2 dataset from the original websites, and set 'image_root' in configs/nlvr.yaml.To evaluate the finetuned BLIP model, runpython -m torch.distributed.run --nproc_per_node=8 train_nlvr.py --evaluateTo finetune the pre-trained checkpoint using 16 A100 GPUs, first set 'pretrained' in configs/nlvr.yaml as "https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model_base.pth". Then run:python -m torch.distributed.run --nproc_per_node=16 train_nlvr.pyFinetune with ViT-L:In order to finetune a model with ViT-L, simply change the config file to set 'vit' as large. Batch size and learning rate may also need to be adjusted accordingly (please see the paper's appendix for hyper-parameter details).Gradient checkpointcan also be activated in the config file to reduce GPU memory usage.Pre-train:Prepare training json files where each json file contains a list. Each item in the list is a dictonary with two key-value pairs: {'image': path_of_image, 'caption': text_of_image}.In configs/pretrain.yaml, set 'train_file' as the paths for the json files .Pre-train the model using 8 A100 GPUs:python -m torch.distributed.run --nproc_per_node=8 pretrain.py --config ./configs/Pretrain.yaml --output_dir output/PretrainZero-shot video-text retrieval:Download MSRVTT dataset following the instructions fromhttps://github.com/salesforce/ALPRO, and set 'video_root' accordingly in configs/retrieval_msrvtt.yaml.Installdecordwithpip install decordTo perform zero-shot evaluation, runpython -m torch.distributed.run --nproc_per_node=8 eval_retrieval_video.pyPre-training datasets download:We provide bootstrapped pre-training datasets as json files. Each json file contains a list. Each item in the list is a dictonary with two key-value pairs: {'url': url_of_image, 'caption': text_of_image}.Image sourceFiltered web captionFiltered synthetic caption by ViT-BFiltered synthetic caption by ViT-LCC3M+CC12M+SBUDownloadDownloadDownloadLAION115MDownloadDownloadDownloadCitationIf you find this code to be useful for your research, please consider citing.@inproceedings{li2022blip,
      title={BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation}, 
      author={Junnan Li and Dongxu Li and Caiming Xiong and Steven Hoi},
      year={2022},
      booktitle={ICML},
}AcknowledgementThe implementation of BLIP relies on resources fromALBEF,Huggingface Transformers, andtimm. We thank the original authors for their open-sourcing.