Package: torchmultimodal-nightly
TorchMultimodal (Beta Release)Models|Example scripts|Getting started|Code overview|Installation|Contributing|LicenseIntroductionTorchMultimodalis a PyTorch library for training state-of-the-art multimodal multi-task models at scale, including both content understanding and generative models. TorchMultimodal contains:A repository of modular and composable building blocks (fusion layers, loss functions, datasets and utilities).A collection of common multimodal model classes built up from said building blocks with pretrained weights for canonical configurations.A set of examples that show how to combine these building blocks with components and common infrastructure from across the PyTorch Ecosystem to replicate state-of-the-art models published in the literature. These examples should serve as baselines for ongoing research in the field, as well as a starting point for future work.ModelsTorchMultimodal contains a number of models, includingALBEF:model class,paperBLIP-2:model class,paperCLIP:model class,paperCoCa:model class,paperDALL-E 2:model,paperFLAVA:model class,paperMAE/Audio MAE:model class,MAE paper,Audio MAE paperMDETR:model class,paperExample scriptsIn addition to the above models, we provide example scripts for training, fine-tuning, and evaluation of models on popular multimodal tasks. Examples can be found underexamples/and includeModelSupported TasksALBEFRetrievalVisual Question AnsweringDDPMTraining and Inference(notebook)FLAVAPretrainingFine-tuningZero-shotMDETRPhrase groundingVisual Question AnsweringMUGENText-to-video retrievalText-to-video generationOmnivorePre-trainingEvaluationGetting startedBelow we give minimal examples of how you can write a simple training or zero-shot evaluation script using components from TorchMultimodal.FLAVA zero-shot exampleimporttorchfromPILimportImagefromtorchmultimodal.models.flava.modelimportflava_modelfromtorchmultimodal.transforms.bert_text_transformimportBertTextTransformfromtorchmultimodal.transforms.flava_transformimportFLAVAImageTransform# Define helper function for zero-shot predictiondefpredict(zero_shot_model,image,labels):zero_shot_model.eval()withtorch.no_grad():image=image_transform(img)["image"].unsqueeze(0)texts=text_transform(labels)_,image_features=zero_shot_model.encode_image(image,projection=True)_,text_features=zero_shot_model.encode_text(texts,projection=True)scores=image_features@text_features.t()probs=torch.nn.Softmax(dim=-1)(scores)label=labels[torch.argmax(probs)]print("Label probabilities: ",{labels[i]:probs[:,i]foriinrange(len(labels))},)print(f"Predicted label:{label}")image_transform=FLAVAImageTransform(is_train=False)text_transform=BertTextTransform()zero_shot_model=flava_model(pretrained=True)img=Image.open("my_image.jpg")# point to your own imagepredict(zero_shot_model,img,["dog","cat","house"])# Example output:# Label probabilities:  {'dog': tensor([0.80590]), 'cat': tensor([0.0971]), 'house': tensor([0.0970])}# Predicted label: dogMAE training exampleimporttorchfromtorch.utils.dataimportDataLoaderfromtorchmultimodal.models.masked_auto_encoder.modelimportvit_l_16_image_maefromtorchmultimodal.models.masked_auto_encoder.utilsimport(CosineWithWarmupAndLRScaling,)fromtorchmultimodal.modules.losses.reconstruction_lossimportReconstructionLossfromtorchmultimodal.transforms.mae_transformimportImagePretrainTransformmae_transform=ImagePretrainTransform()dataset=MyDatasetClass(transforms=mae_transform)# you should define thisdataloader=DataLoader(dataset,batch_size=8)# Instantiate model and lossmae_model=vit_l_16_image_mae()mae_loss=ReconstructionLoss()# Define optimizer and lr scheduleroptimizer=torch.optim.AdamW(mae_model.parameters())lr_scheduler=CosineWithWarmupAndLRScaling(optimizer,max_iters=1000,warmup_iters=100# you should set these)# Train one epochforbatchindataloader:model_out=mae_model(batch["images"])loss=mae_loss(model_out.decoder_pred,model_out.label_patches,model_out.mask)loss.backward()optimizer.step()lr_scheduler.step()Code overviewtorchmultimodal/diffusion_labsdiffusion_labs contains components for building diffusion models. For more details on these components, seediffusion_labs/README.md.torchmultimodal/modelsLook here for model classes as well as any other modeling code specific to a given architecture. E.g. the directorytorchmultimodal/models/blip2contains modeling components specific to BLIP-2.torchmultimodal/modulesLook here for common generic building blocks that can be stitched together to build a new architecture. This includeslayerslikecodebooks,patch embeddings, ortransformer encoder/decoders,losseslikecontrastive loss with temperatureorreconstruction loss,encoderslikeViTandBERT, andfusion moduleslikeDeep Set fusion.torchmultimodal/transformsLook here for common data transforms from popular models, e.g.CLIP,FLAVA, andMAE.InstallationTorchMultimodal requires Python >= 3.8. The library can be installed with or without CUDA support.
The following assumes conda is installed.PrerequisitesInstall conda environmentconda create -n torch-multimodal python=\
conda activate torch-multimodalInstall pytorch, torchvision, and torchaudio. SeePyTorch documentation.# Use the current CUDA version as seen [here](https://pytorch.org/get-started/locally/)
# Select the nightly Pytorch build, Linux as the OS, and conda. Pick the most recent CUDA version.
conda install pytorch torchvision torchaudio pytorch-cuda=\ -c pytorch-nightly -c nvidia

# For CPU-only install
conda install pytorch torchvision torchaudio cpuonly -c pytorch-nightlyInstall from binariesNightly binary on Linux for Python 3.8 and 3.9 can be installed via pip wheels.
For now we only support Linux platform throughPyPI.python -m pip install torchmultimodal-nightlyBuilding from SourceAlternatively, you can also build from our source code and run ourexamples:git clone --recursive https://github.com/facebookresearch/multimodal.git multimodal
cd multimodal

pip install -e .For developers please follow thedevelopment installation.ContributingWe welcome any feature requests, bug reports, or pull requests from the community. See theCONTRIBUTINGfile for how to help out.LicenseTorchMultimodal is BSD licensed, as found in theLICENSEfile.