Package: pali-torch
PALI: A JOINTLY-SCALED MULTILINGUAL LANGUAGE-IMAGE MODELThe open source implementation of the Multi-Modality AI model from"PaLI: Scaling Language-Image Learning in 100+ Languages"NOTEThis is the base model architecture, no tokenizer or pretrained weightsTo train, find tokenizer, like tokenmonster and patchify the images to make it compatible with example.pyWe're utilizing an Encoder/Decoder as UL2 and a VIT model that embeds the image which is then injected into the text encoder decoderIf you would like to help train this model and release it open source please click on the agora banner and join the lab!🌟 AppreciationBig bear hugs 🐻💖 toLucidRainsfor the fab x_transformers and for championing the open source AI cause.🚀 Quick Startpipinstallpali-torch🧙 UsageimporttorchfrompaliimportPalimodel=Pali()img=torch.randn(1,3,256,256)prompt=torch.randint(0,256,(1,1024))mask=torch.ones(1,1024).bool()output_text=torch.randint(0,256,(1,1024))result=model.forward(img,prompt,output_text,mask)print(result)Datasets StrategyDataset strategy as closely shown in the paper.Here is a markdown table with metadata and links to the datasets on HuggingFace for the datasets used:DatasetDescriptionSizeLanguagesLinkWebLILarge-scale web crawled image-text dataset10B images, 12B captions109 languagesPrivateCC3MConceptual Captions dataset3M image-text pairsEnglishLinkCC3M-35LTranslated version of CC3M to 35 languages105M image-text pairs36 languagesPrivateVQAv2VQA dataset built on COCO images204K images, 1.1M QA pairsEnglishLinkVQ2A-CC3MVQA dataset built from CC3M3M image-text pairsEnglishPrivateVQ2A-CC3M-35LTranslated version of VQ2A-CC3M to 35 languages105M image-text pairs36 languagesPrivateOpen ImagesLarge scale image dataset9M images with labelsEnglishLinkVisual GenomeImage dataset with dense annotations108K images with annotationsEnglishLinkObject365Image dataset for object detection500K images with labelsEnglishPrivateThe key datasets used for pre-training PaLI include:WebLI: A large-scale multilingual image-text dataset crawled from the web, comprising 10B images and 12B captions in 109 languages.CC3M-35L: CC3M Conceptual Captions dataset machine translated into 35 additional languages, totaling 105M image-text pairs in 36 languages.VQ2A-CC3M-35L: VQA dataset based on CC3M, also translated into 35 languages.The model was evaluated on diverse tasks using standard datasets like VQAv2, Open Images, COCO Captions etc. Links and details provided above.🎉 FeaturesDouble the Power: MT5 for text and ViT for images - Pali's the superhero we didn't know we needed! 💪📖🖼️Winning Streak: With roots in the tried-and-true MT5 & ViT, success is in Pali's DNA. 🏆Ready, Set, Go: No fuss, no muss! Get Pali rolling in no time. ⏱️Easy-Peasy: Leave the heavy lifting to Pali and enjoy your smooth sailing. 🛳️🌆 Real-World Use-CasesE-commerce: Jazz up those recs! Understand products inside-out with images & descriptions. 🛍️Social Media: Be the smart reply guru for posts with pics & captions. 📱Healthcare: Boost diagnostics with insights from images & textual data. 🏥📚 Citation@inproceedings{chen2022pali,
  title={PaLI: Scaling Language-Image Learning in 100+ Languages},
  author={Chen, Xi and Wang, Xiao},
  booktitle={Conference on Neural Information Processing Systems (NeurIPS)},
  year={2022}
}TodoMake a table of datasets used in paper,Provide tokenizer integrationProvide training scriptProvide usage/inference scripts📜 LicenseMIT