Package: clip-benchmark
CLIP BenchmarkThe goal of this repo is to evaluate CLIP-like models on a standard set
of datasets on different tasks such as zero-shot classification and zero-shot
retrieval, and captioning.Below we show the average rank (1 is the best, lower is better) of different CLIP models, evaluated
on different datasets.The current detailed results of the benchmark can be seenhereor directly in thenotebook.FeaturesSupport for zero-shot classification and zero-shot retrieval, linear probing, and captioning.Support forOpenCLIPpre-trained models,Japanese CLIP, andNLLB CLIPfor general multilingual abilities.Support various datasets fromtorchvision,tensorflow datasets, andVTAB.Support for various multilingual datasets for classification and retrievalSupport for compositionality tasksHow to install?pip install clip-benchmarkHow to use?To evaluate we recommend to create a models.txt likeViT-B-32,openaito get the list of datasetswget https://raw.githubusercontent.com/LAION-AI/CLIP_benchmark/main/benchmark/webdatasets.txtThen to runclip_benchmark eval --pretrained_model models.txt \
    --dataset "webdatasets.txt" \
    --dataset_root "https://huggingface.co/datasets/clip-benchmark/wds_{dataset_cleaned}/tree/main" \
    --output "benchmark_{dataset}_{pretrained}_{model}_{language}_{task}.json"Then to get the full tableclip_benchmark build benchmark_*.json --output benchmark.csvCommand line interface (CLI)The easiest way to benchmark the models is using the CLI,clip_benchmark.
You can specify the model to use, the dataset and the task to evaluate on. Once it is done, evaluation is performed and
the results are written into a JSON file.Using other models than openclipIt is possible to use other models than openclip ones. For example japanese-clip is supportedHere is an example of use>>> python3 clip_benchmark/cli.py eval \
  --model_type "ja_clip" \ # flag to use japanese-clip
  --pretrained "rinna/japanese-cloob-vit-b-16" \ # now, we have `rinna/japanese-cloob-vit-b-16` or `rinna/japanese-clip-vit-b-16`. 
  --language "jp" \
  --task "zeroshot_classification"  \
  --dataset "imagenet1k"  \
  --dataset_root {ROOT_PATH} 

>>> cat result.json
{"dataset": "imagenet1k", "model": "ViT-B-32-quickgelu", "pretrained": "rinna/japanese-cloob-vit-b-16", "task": "zeroshot_classification", "metrics": {"acc1": 0.54636, "acc5": 0.72856, "mean_per_class_recall": 0.54522}, "language": "jp"}How to add other CLIP modelsPlease follow these steps:Add a identity file to load model inclip_benchmark/modelsDefine a loading function, that returns a tuple (model, transform, tokenizer). Please seeclip_benchmark/models/open_clip.pyas an example.Add the function intoTYPE2FUNCinclip_benchmark/models/__init__.pyRemarks:The new tokenizer/model must enable to do the following things ashttps://github.com/openai/CLIP#usagetokenizer(texts).to(device)...textsis a list of stringmodel.encode_text(tokenized_texts)...tokenized_textsis a output fromtokenizer(texts).to(device)model.encode_image(images)...imagesis a image tensor by thetransformCIFAR-10 exampleHere is an example for CIFAR-10 zero-shot classification using OpenCLIP's pre-trained model on LAION-400m:clip_benchmark eval --dataset=cifar10 --task=zeroshot_classification --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64By default, the dataset is downloaded into--dataset_root, which by default isroot.Here is the content ofresult.jsonafter the evaluation is done:{"dataset":"cifar10","model":"ViT-B-32-quickgelu","pretrained":"laion400m_e32","task":"zeroshot_classification","metrics":{"acc1":0.9074,"acc5":0.998}}VOC2007 exampleHere is another example with VOC2007, which is a multi-label classification dataset.clip_benchmark eval --dataset=voc2007_multilabel --task=zeroshot_classification --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64Here is the content ofresult.jsonafter the evaluation is done:{"dataset":"voc2007_multilabel","model":"ViT-B-32-quickgelu","pretrained":"laion400m_e32","task":"zeroshot_classification","metrics":{"mean_average_precision":0.7627869844436646}}Here, we compute the mean average precision or mAP, more details about that metricherein the context of multi-label classification.VTAB exampleHere is an example on how to run it onVTABclassification tasks.
First, you need to install VTAB's dedicated package.pip install task_adaptation==0.1Then, you can run it by providing the full dataset name.
Example witheurosat:clip_benchmark eval --dataset=vtab/eurosat --task=zeroshot_classification --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64Seeclip_benchmark/datasets/builder.py#L634for the full list of
VTAB dataset collection.TensorFlow dataset exampleHere is an example on how to run it onTensorflow datasets.
First, you need to installtfds-nightlyandtimm.pip install timm tfds-nightlyThe name of the dataset follows the templatetfds/<DATASET_NAME>.Example withcifar10:clip_benchmark eval --dataset=tfds/cifar10 --task=zeroshot_classification --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64COCO captions retrieval exampleHere is an example for COCO captions zero-shot retrieval:clip_benchmark eval --dataset=mscoco_captions --task=zeroshot_retrieval --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64Note that for using COCO, you also need to installpycocotools(e.g., usingpip install pycocotools).COCO captions captioning exampleHere is an example for COCO captions captioning task:clip_benchmark eval --dataset=mscoco_captions --task=captioning --model=coca_ViT-L-14 --output=result.json --pretrained mscoco_finetuned_laion2b_s13b_b90kNote that for using COCO, you also need to installpycocotools(e.g., usingpip install pycocotools) andpycocoevalcap.Linear probing exampleFull linear probing on train split, evaluate on test split:clip_benchmark eval --dataset=cifar10 --task=linear_probe --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64 --fewshot_lr 0.1 --fewshot_epochs 20 --batch_size 512 --train_split train --test_split testfew-shot (k=5) linear probing on train split, evaluate on test split:clip_benchmark eval --dataset=cifar10 --task=linear_probe --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64 --fewshot_k 5 --fewshot_lr 0.1 --fewshot_epochs 20 --batch_size 512 --train_split train --test_split testSplit train into train (90%), val (10%), do linear probing on train split, tune on val split, evaluate on test split:clip_benchmark eval --dataset=cifar10 --task=linear_probe --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64 --fewshot_lr 0.1 --fewshot_epochs 20 --batch_size 512 --train_split train --val_proportion 0.1 --test_split testFor other datasets that have an official val split, one can also specify the val split:clip_benchmark eval --dataset=fgvc_aircraft --task=linear_probe --pretrained=laion400m_e32 --model=ViT-B-32-quickgelu --output=result.json --batch_size=64 --fewshot_lr 0.1 --fewshot_epochs 20 --batch_size 512 --train_split train --val_split val --test_split testMultilingual evaluationWe also provide datasets for evaluating multilingual models (see e.g.https://github.com/mlfoundations/open_clip#vit-b32-xlm-roberta-base, andhttps://github.com/mlfoundations/open_clip/blob/main/docs/openclip_multilingual_retrieval_results.csv) by specifying--language.For ImageNet-1k (zero-shot classification):clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=imagenet1k --output=result.json --batch_size=64 --language=<LANG>, where<LANG>can be amongzh(chinese),it(italian),jp(japanese),en(english),ar(arabic).We also support Babel ImageNet classnames and prompts (https://github.com/gregor-ge/Babel-ImageNet), which can be used as the following:clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=babel_imagenet --output=result.json --batch_size=64 --language=<LANG>,
where<LANG>is a two letter string from theISO language code list. Supported values for language are:'ne', 'id', 'de', 'nl', 'af', 'he', 'sq', 'uz', 'kn', 'ku', 'ta', 'lv', 'ko', 'ug', 'br', 'el', 'su', 'kk', 'sk', 'gl', 'om', 'fa', 'jv', 'cs', 'lo', 'hy', 'xh', 'hr', 'so', 'gu', 'am', 'ar', 'sa', 'ca', 'is', 'it', 'sv', 'ga', 'bg', 'vi', 'sd', 'ur', 'km', 'pl', 'hu', 'sr', 'fr', 'hi', 'fy', 'et', 'bs', 'sw', 'az', 'mk', 'es', 'mn', 'ja', 'tl', 'tr', 'gd', 'ro', 'mg', 'mr', 'sl', 'pt', 'lt', 'no', 'yi', 'uk', 'ky', 'ka', 'bn', 'or', 'my', 'en', 'ps', 'fi', 'zh', 'da', 'ml', 'be', 'eo', 'ha', 'eu', 'as', 'te', 'th', 'cy', 'si', 'ru', 'la', 'pa', 'ms'for COCO (zero-shot retrieval):clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=multilingual_mscoco_captions --output=result.json --batch_size=64 --language=<LANG>, where<LANG>can be amonges(spanish),it(italian),jp(japanese),ko(korean),pl(polish),ru(russian),tr(Turkish),zh(chinese),en(english),fr(french),de(german).For Flickr-30k (zero-shot retrieval)clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=flickr30k --output=result.json --batch_size=64 --language=<LANG>, where<LANG>can be amongen(english),zh(chinese).For Flickr-8k (zero-shot retrieval)clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=flickr8k --output=result.json --batch_size=64 --language=<LANG>, where<LANG>can be amongen(english),zh(chinese).ForCrossmodal-3600(zero-shot retrieval)clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=crossmodal3600 --output=result.json --batch_size=64 --language=<LANG>, see supported languageshere.For Flickr30k-200 dataset, which has 1000 captions from Flickr30k dataset translated to 200 languages usingNLLB-3.3B model(zero-shot retrieval)clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=flickr30k-200 --output=result.json --batch_size=64 --language=<LANG>, see supported languageshere.For XTD200 dataset, which has captions fromXTD10dataset translated to 200 languages usingNLLB-3.3B model(zero-shot retrieval)clip_benchmark eval --model xlm-roberta-base-ViT-B-32 --pretrained laion5b_s13b_b90k --dataset=xtd200 --output=result.json --batch_size=64 --language=<LANG>, see supported languageshere.Compositionality evaluationForSugar Crepe:clip_benchmark eval --model ViT-B-32 --pretrained laion400m_e32 --dataset=sugar_crepe/<TASK> --output=result.jsonwhere<TASK>can be amongadd_att,add_obj,replace_att,replace_obj,replace_rel,swap_att,swap_obj.
To evaluate on all the tasks together, you can do:clip_benchmark eval --model ViT-B-32 --pretrained laion400m_e32 --dataset=sugar_crepe --output=result.jsonForwinoground:clip_benchmark eval --model ViT-B-32 --pretrained laion400m_e32 --dataset=winoground --output=result.jsonNB:pip install datasetsis required for winoground.Webdataset exampleHere is an example on how to run it onwebdatasets.
First, you need to installwebdataset.pip install webdatasetCreating a webdatasetYou can either convert an already supported CLIP_benchmark dataset to webdataset format, or manually create your own with the same file structure. For already supported datasets use the CLI commandclip_benchmark_export_wdsas in this example:$ clip_benchmark_export_wds --dataset cifar10 --plit train --dataset_root DATA_DIR/ --output wds_cifar10/
$ clip_benchmark_export_wds --dataset cifar10 --split test --dataset_root DATA_DIR/ --output wds_cifar10/which will convert the train and test splits for CIFAR-10 (downloaded toDATA_DIR/) and save the webdataset towds_cifar10/(upload to Huggingface Hub must be done manually for now). Retrieval datasets are also supported with the--retrievalflag.For other datasets, data must be stored with the following file structure:root_dir/
    train/
        nshards.txt
        0.tar
        1.tar
        ...
    test/
        nshards.txt
        0.tar
        ...
    classnames.txt
    zeroshot_classification_templates.txt
    dataset_type.txtEach split should be contained in its own folder andnshards.txtshould contain a single integer corresponding to the number of TAR files. The TAR files should follow webdataset format, with an image file (.webp, .png, or .jpg) and a label (.cls) for each example. Classnames and templates are required for zeroshot classification evaluation, with each classname or template on its own line. Dataset type is required for distinguishing zeroshot retrieval evaluation: the file should just contain the textretrieval.Evaluating on a webdatasetThe name of the dataset follows the templatewds/<DATASET_NAME>. Note that the dataset name currently only affects the name in the results output - classnames and templates are loaded directly from the included files. The dataset root directory can be either a local path to theroot_diras specified above, or an HTTP URL pointing to a Huggingface Hub dataset file tree.Example withvtab/cifar10(zero-shot classification):local:clip_benchmark eval --dataset wds/vtab/cifar10 --dataset_root ROOT_DIR/wds_vtab-cifar10/remote:clip_benchmark eval --dataset wds/vtab/cifar10 --dataset_root https://huggingface.co/datasets/clip-benchmark/wds_{dataset_cleaned}/tree/mainExample withmscoco_captions(retrieval):local:clip_benchmark eval --dataset=wds/mscoco_captions --dataset_root ROOT_DIR/wds_vtab-mscoco_captions/ --task=zeroshot_retrievalremote:clip_benchmark eval --dataset=wds/mscoco_captions --dataset_root="https://huggingface.co/datasets/clip-benchmark/wds_{dataset_cleaned}/tree/main" --task=zeroshot_retrievalExample withmscoco_captions(captioning):local:clip_benchmark eval --dataset=wds/mscoco_captions --dataset_root ROOT_DIR/wds_vtab-mscoco_captions/ --task=captioningremote:clip_benchmark eval --dataset=wds/mscoco_captions --dataset_root="https://huggingface.co/datasets/clip-benchmark/wds_{dataset_cleaned}/tree/main" --task=captioningAll other arguments remain the same as in the other examples. Seehttps://huggingface.co/clip-benchmarkfor a full list of datasets that have already been uploaded to Huggingface.Evaluate mulitple models on multiple datasetsFor the purpose of benchmarking, it is possible to run the CLI with multiple
pre-trained models on multiple datasets.Pretrained models and datasets list as argumentsFor models, we can provide list of pretrained model names in the form of 'model,pretrained' (somodelandpretrainedare comma separated). For datasets, we can provide a list of datasets.  For languages, we can provide a list of languages.
Example:clip_benchmarkeval--pretrained_modelViT-B-32-quickgelu,laion400m_e32ViT-L-14,laion400m_e32\--datasetcifar10cifar100--dataset_root"clip_benchmark_datasets/{dataset}"--languageenjp\--output"{dataset}_{pretrained}_{model}_{language}_{task}.json"Note that--dataset_rootand--outputcan be now in the form of a template that depends on the dataset/model/language/task (for--output) and dataset name (for--dataset_root).Note that If the benchmark fails at some point, it is possible to resume it by skipping already evaluated models using--skip_existing.Pretrained models and datasets list as filesWe can also provide a path to files with models (each line is in the form of 'model,pretrained' wheremodelandpretrainedare comma separated) and datasets list (one dataset per line):clip_benchmarkeval--pretrained_modelbenchmark/models.txt\--datasetbenchmark/datasets.txt--dataset_root"clip_benchmark_datasets/{dataset}"\--output"{dataset}_{pretrained}_{model}_{language}_{task}.json"Examples are available inbenchmark/datasets.txtandbenchmark/models.txtMultiple checkpoints from the same modelIt is also common to evaluate multiple checkpoints from the same model:clip_benchmarkeval--modelViT-B-32--pretrained*.pt\--datasetbenchmark/datasets.txt--dataset_root"clip_benchmark_datasets/{dataset}"\--output"{dataset}_{pretrained}_{model}_{language}_{task}.json"Model and dataset collectionsWe can also provide model collection names (openai,openclip_base,openclip_multilingual,openclip_fullare supported) or dataset collection names (vtab,vtab+,retrieval,imagenet_robustnessare supported):clip_benchmarkeval--pretrained_modelopenaiopenclip_base--datasetvtab+retrieval\--dataset_root"clip_benchmark_datasets/{dataset}"--notquiet\--output"{dataset}_{pretrained}_{model}_{language}_{task}.json"Seeclip_benchmark/models.py#L6andclip_benchmark/datasets/builder.py#L634for more information
about the collections.Custom templates / prompts / classnamesIt is also possible to use custom prompts by providing a custom template file and/or a custom classname file.
For instance:clip_benchmark eval --dataset "imagenet1k" --model ViT-B-32 --pretrained laion400m_e32 --custom_template_file <path> --custom_classname_file <path>The template file can be either in the usual formathttps://github.com/LAION-AI/CLIP_benchmark/blob/main/clip_benchmark/datasets/en_zeroshot_classification_templates.jsonor in the CuPL formathttps://github.com/LAION-AI/CLIP_benchmark/blob/main/clip_benchmark/datasets/cupl_prompts.jsonto have class-specific prompts. In the case of the CuPL format, the classnames file will not be used, thus one only needs to provide the template file--custom_template_file.For instance, the prompts from the CuPL paperhttps://arxiv.org/abs/2209.03320for ImagetNet-1k can be used this way    :clip_benchmark eval --dataset "imagenet1k" --model ViT-B-32 --pretrained laion400m_e32 --custom_template_file cupl_prompts.jsonDevelopmentFor development, you can also do this:gitclonehttps://github.com/LAION-AI/CLIP_benchmarkcdCLIP_benchmark
pythonsetup.pyinstallCreditsThanks toOpenCLIPauthors, zero-shot accuracy code is adapted from there and pre-trained models are used in the command line interface.Thanks toSLIPauthors, some zero-shot templates and classnames are from there.Thanks toWise-ftauthors, Imagenet robustness datasets code is adapted from thereThanks toLiTauthors, some zero-shot templates and classnames of VTAB datasets are from there.Thanks toSugar Crepeauthors for compositionality tasks evaluation on COCOThanks toBabel ImageNetauthors for multilingual evaluation of ImageNet-1k zero-shot classification.Thanks toImageNet-Wauthors for ImageNet-W evaluationThanks toCuPLfor CuPL prompts.Thanks toPyCOCOevalcapand@gpuccefor COCO captions image captioning evaluation.Thanks to@li-xirongfor chinese Flickr-30k/FLickr-8k.Thanks toChinese CLIPauthors for chinese ImageNet-1k classnames/prompts (zero-shot classification).Thanks to@rinnakkand@mkshingfor japanese ImageNet-1k classnames/prompts (zero-shot classification) and japanese CLIP support.Thanks to@KhalidAltfor arabic ImageNet-1k classnames/prompts (zero-shot classification).Thanks to@djghosh13for WebDataset support.Thanks to@FreddeFrallanfor for multilingual COCO.Thanks to@mitchellnwfor linear probing support.Thanks to@teasgenfor support of validation set and tuning linear probing similar to OpenAI's CLIP.Thanks to@visheratinfor multilingual retrieval datasets support fromhttps://arxiv.org/abs/2309.01859.This package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template. Thanks to the author.History1.6.1Fix missing sugar crepe example #119 thanks to @samarth41491.6.0Fix overwritten zeroshot templates issue (https://github.com/LAION-AI/CLIP_benchmark/issues/109)Support new multilingual retrieval datasets:  Crossmodal-3600, XTD10, Flickr30k-200, and XTD200Support tuning linear probing on validation set1.5.0Custom classnames and templatessupport wds for captioning evaluationsupport imagenet-wsupport babel imagenetsupport chinese flickr30k/8ksupport sugar crepe (compositionality)support (optional) sharding evaluation based on rank, for parallel runsfix many issues1.4.0Fix silent webdataset error-handlingAdded support for wds/voc2007_multilabeldefault to float32add mscoco generative benchmark1.3.0update flickr8k results, solve issue #48, thanks to @orchidmajumderEvaluate multiple models/datasets/languages using the CLI directlySupport Japanese CLIP by rinnaAdd arabic imagenetupdating CuPL prompts with more generated sentences + ensembled with openAI promptsput model in eval mode before evaluationWebdataset updatesMake verbose the default1.2.0Added support for loading webdatasets1.1.0Added better support for multilingual evalAdded better support for linear probingAdded support for CuPL prompts1.0.1pypi description as markdown1.0.0Actual first release on PyPI.0.1.0First release on PyPI.