Package: mclip-anytorch
Multilingual-CLIPOpenAI CLIP text encoders for any languageLive Demo·Pre-trained Models·Report BugOverviewOpenAIrecently released the paperLearning Transferable Visual Models From Natural Language Supervisionin which they present the CLIP (Contrastive Language–Image Pre-training) model. This model is trained to connect text and images, by matching their corresponding vector representations using a contrastive learning objective.
CLIP consists of two separate models, a visual encoder and a text encoder. These were trained on a wooping 400 Million images and corresponding captions.
OpenAI has since released a set of their smaller CLIP models, which can be found on theofficial CLIP Github.DemoA live demonstration of multilingual Text-Image retrieval using M-CLIP can be foundhere!This demo was created byRom1504, and it allows you to search the LAION-400M dataset in various languages using M-CLIP.This repository containsPre-trained CLIP-Text encoders for multiple languagesPytorch & Tensorflow inference codeTensorflow training codeRequirementsWhile it is possible that other versions works equally fine, we have worked with the following:Python = 3.6.9Transformers = 4.8.1Installpip install multilingual-clip torchYou can also choose topip install tensorflowinstead of torch.Inference UsageInference code for Tensorflow is also available ininference_example.pyfrommultilingual_clipimportpt_multilingual_clipimporttransformerstexts=['Three blind horses listening to Mozart.','Älgen är skogens konung!','Wie leben Eisbären in der Antarktis?','Вы знали, что все белые медведи левши?']model_name='M-CLIP/XLM-Roberta-Large-Vit-L-14'# Load Model & Tokenizermodel=pt_multilingual_clip.MultilingualCLIP.from_pretrained(model_name)tokenizer=transformers.AutoTokenizer.from_pretrained(model_name)embeddings=model.forward(texts,tokenizer)print(embeddings.shape)Install for developmentSetup a virtualenv:python3 -m venv .env
source .env/bin/activate
pip install -e .Pre-trained ModelsEvery text encoder is aHuggingfaceavailable transformer, with an additional linear layer on top. For more information of a specific model, click the Model Name to see its model card.NameModel BaseVision ModelVision DimensionsPre-trained Languages#ParametersLABSE Vit-L/14LaBSEOpenAI ViT-L/14768109 Languages110 MXLM-R Large Vit-B/32XLM-Roberta-LargeOpenAI ViT-B/32512100 Languages344 MXLM-R Large Vit-L/14XLM-Roberta-LargeOpenAI ViT-L/14768100 Languages344 MXLM-R Large Vit-B/16+XLM-Roberta-LargeOpen CLIP ViT-B-16-plus-240640100 Languages344 MValidation & Training CurvesFollowing is a table of theTxt2Img @10-Recalfor the humanly tanslatedMS-COCO testset.NameEnDeEsFrZhItPlKoRuTrJpOpenAI CLIP Vit-B/3290.3----------OpenAI CLIP Vit-L/1491.8----------OpenCLIP ViT-B-16+-94.3----------LABSE Vit-L/1491.689.689.589.988.990.189.880.885.589.873.9XLM-R Large Vit-B/3291.888.789.189.489.389.891.482.186.188.881.0XLM-R Vit-L/1492.490.691.090.089.791.191.385.285.890.381.9XLM-R Large Vit-B/16+95.093.093.693.194.093.194.489.090.093.084.2The training curves for these models are available at thisWeights and Biases Report, the results for other non-succesfull and ongoing experiments can be found in theWeights and Biases Project.Legacy Usage and ModelsOlder versions of M-CLIP had the linear weights stored separately from Huggingface. Whilst the new models have them directly incorporated in the Huggingface repository. More information about these older models can be found in this section.Click for more informationDownload CLIP Model$condainstall--yes-cpytorchpytorch=1.7.1torchvisioncudatoolkit=11.0
$pipinstallftfyregextqdm
$pipinstallgit+https://github.com/openai/CLIP.gitReplacecudatoolkit=11.0above with the appropriate CUDA version on your machine orcpuonlywhen installing on a machine without a GPU.
For more information please see the officialCLIP repostitory.Download Linear Weights# Linear Model Weights$bashlegacy_get-weights.shInferencefrommultilingual_clipimportmultilingual_clipprint(multilingual_clip.AVAILABLE_MODELS.keys())model=multilingual_clip.load_model('M-BERT-Distil-40')embeddings=model(['Älgen är skogens konung!','Wie leben Eisbären in der Antarktis?','Вы знали, что все белые медведи левши?'])print(embeddings.shape)# Yields: torch.Size([3, 640])For a more elaborate example, comparing the textual embeddings to the CLIP image embeddings see thiscolab notebook.Legacy Pre-trained ModelsEvery text encoder is aHuggingfaceavailable transformer, with an additional linear layer on top. Neither of the models have been extensively tested, but for more information and qualitative test results for a specific model, click the Model Name to see its model card.*** Make sure to update to the most recent version of the repostitory when downloading a new model, and re-run the shell script to download the Linear Weights. ***NameModel BaseVision ModelPre-trained LanguagesTarget Languages#ParametersMultilingualM-BERT Distil 40M-BERT DistilRN50x4101 Languages40 Languages66 MM-BERT Base 69M-BERT BaseRN50x4101 Languages68 Languages110 MM-BERT Base ViT-BM-BERT BaseViT-B/32101 Languages68 Languages110 MMonolingualSwe-CLIP 500kKB-BERTRN50x4SwedishSwedish110 MSwe-CLIP 2MKB-BERTRN50x4SwedishSwedish110 MTraining a new modelThis foldercontains the code used for training the above models. If you wsh to train your own model you must do the following things:Prepare a set of translated sentence pairs from English -> Your Language(s)Compute regular CLIP-Text embeddings for the English sentences.EditTraining.pyto load your data.Train a new CLIP-Text encoder via Teacher LearningPre-computed CLIP Embeddings & Translaton DataThis Google Drive foldercontains both pre-computed CLIP-Text Embeddings for a large porton of the the image captions ofGCC+MSCOCO+VizWiz.The Google Drive folder also contains the translation data used to train the currently available models.
Good LuckContributionIf you have trained a CLIP Text encoder specific to your language, or another model covering a language not supported here, Please feel free to contact us and we will either upload your model and credit you, or simply link to your already uploaded model.ContactIf you have questions regarding the code or otherwise related to this Github page, please open anissue.For other purposes, feel free to contact me directly at:Fredrik.Carlsson@ri.seAcknowledgementsStability.aifor providing much appreciated compute during training.CLIPOpenAIHuggingfaceBest Readme Template"Two Cats" Image by pl1602LicenseDistributed under the MIT License. SeeLICENSEfor more information.CitingIf you found this repository useful, please consider citing:@InProceedings{carlsson-EtAl:2022:LREC,author={Carlsson, Fredrik  and  Eisen, Philipp  and  Rekathati, Faton  and  Sahlgren, Magnus},title={Cross-lingual and Multilingual CLIP},booktitle={Proceedings of the Language Resources and Evaluation Conference},month={June},year={2022},address={Marseille, France},publisher={European Language Resources Association},pages={6848--6854},abstract={The long-standing endeavor of relating the textual and the visual domain recently underwent a pivotal breakthrough, as OpenAI released CLIP. This model distinguishes how well an English text corresponds with a given image with unprecedented accuracy. Trained via a contrastive learning objective over a huge dataset of 400M of images and captions, it is a work that is not easily replicated, especially for low resource languages. Capitalizing on the modularization of the CLIP architecture, we propose to use cross-lingual teacher learning to re-train the textual encoder for various non-English languages. Our method requires no image data and relies entirely on machine translation which removes the need for data in the target language. We find that our method can efficiently train a new textual encoder with relatively low computational cost, whilst still outperforming previous baselines on multilingual image-text retrieval.},url={https://aclanthology.org/2022.lrec-1.739}}