Package: vl-checklist
VL-CheckList (Coming Soon)Updates07/04/2022: VL-CheckList paper on arxivhttps://arxiv.org/abs/2207.0022107/12/2022: Updated object, relation, attribute splits/dataset08/01/2022: Release the initial code and an example modelIntroductionThis repository is the official project page for (VL-CheckList).VL-CheckListis an explainable framework that comprehensively evaluates VLP models and facilitates deeper understanding.The current method to evaluate a VLP model is solely by comparing its fine-tuned downstream tasks performance, which has a number of limitations, such as poor interpretability, incomparable results and bias in data.The core principle of VL-CheckList are: (1) evaluate a VLP model's fundamental capabilities instead of performance on applications (2) disentangle capabilities into relatively independent variables that are easier to analyze.VL-CheckList evaluates VLP models from three aspects: Object, Attribute and Relationship. We provide the performance quantitative table and the radar chart based on the three aspects.How to Install VL-CheckListThere are two options to use VL-CheckList. First, you can install vl_checklist in your project and import vl_checklist and evaluate your modelspip install vl_checklistSecond, you can clone the project add your model engine which include 'predict' function. Please find an example engineLinkgit clone https://github.com/om-ai-lab/VL-CheckList.gitHow to Evaluate your ModelWe put the example models in theexample_models/folder.1.Define a config file
e.g. inconfigs/sample.yamMAX_NUM: 2000
MODEL_NAME: "ViLT"
BATCH_SIZE: 4
TASK: "itc"
DATA:
  TYPES: ["Attribute/color"]
  TEST_DATA: ["vg","vaw"]   
OUTPUT: 
  DIR: "output/vilt"2.Prepare Evaluation Data
We provide the initial curated jsons atdata/and corresponding yamls atvl_checklist/corpus. You can need to download image dataset. You can find the instruction in detailLink3.Load the model which containpredict()and Evaluate class as follows.4.Runstart()as followsHere is an example codefromexample_models.vilt.engineimportViLTfromvl_checklist.evaluateimportEvaluateif__name__=='__main__':model=ViLT('vilt_200k_mlm_itm.ckpt')vilt_eval=Evaluate(config="configs/sample.yaml",model=model)vilt_eval.start()5.check the results in the OUTDIR DIR you defined the yaml file
You can check the output formatLINKDownload Pretrained WeightsWe include examples models atexample_models/. You can download the pretrained weights we used in our papers as follows:ViLT-B/32ALBEFTCLUNITEROSCARLXMERTCLIPDemoWe present the demo in huggingface space, you can try it here:Demo linkIn this demo, you can change the object and attribute of object in the text prompt. You can also change the size and location of the object.ReferencesIf you use any source codes or datasets included in this toolkit in your work, please cite the following paper. The bibtex are listed below:@misc{https://doi.org/10.48550/arxiv.2207.00221,
  doi = {10.48550/ARXIV.2207.00221}, 
  url = {https://arxiv.org/abs/2207.00221},
  author = {Zhao, Tiancheng and Zhang, Tianqi and Zhu, Mingwei and Shen, Haozhan and Lee, Kyusong and Lu, Xiaopeng and Yin, Jianwei},
  keywords = {Computer Vision and Pattern Recognition (cs.CV), Computation and Language (cs.CL), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations},
  publisher = {arXiv},
  year = {2022},
  copyright = {Creative Commons Attribution 4.0 International}
}