Package: m2pt
Multi-Modal Pathway TransformerImplementation of M2PT in PyTorch from the paper: "Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities".PAPER LINK. This is really really cool because just by merging the projections of different multi-modal models together you can increase the performance of your base model. This is a small but effective technique that can be implemented in any model with a minor plug in.Installpip3 install -U m2ptUsageM2PTA fully ready to train implementation of the M2PT model that can be merged with the linears from any multi-modal models, just plug it in! It takes in tokenized texts which are integers then embeds them and then passes -> them into the transformer blocks and then at the end projects them and applies a softmaximporttorchfromtorchimportnnfromm2pt.mainimportM2PT# Create an instance of the M2PT model class with the specified parametersmodel=M2PT(dim=512,# Dimension of the input and output tensorsnum_tokens=10000,depth=6,dim_head=64,# Dimension of each attention headheads=8,# Number of attention headsdropout=0.1,# Dropout rateff_mult=4,# Multiplier for the dimension of the feed-forward networkoriginal_linear=nn.Linear(512,512),# Linear layer for the original input tensorauxiliar_linear=nn.Linear(512,512),# Linear layer for the auxiliary input tensorffn_original_linear=nn.Linear,# Linear layer for the original input tensor in the feed-forward networkffn_auxiliar_linear=nn.Linear,# Linear layer for the auxiliary input tensor in the feed-forward networkffn_original_last_linear=nn.Linear,# Last linear layer for the original input tensor in the feed-forward networkffn_aux_last_linear=nn.Linear,# Last linear layer for the auxiliary input tensor in the feed-forward network)# Create a 3D tensor with shape B x S x Dx=torch.randint(0,10000,(1,512))# Pass the input tensor through the modelout=model(x)# Print the shape of the output tensorprint(out.shape)MPTransformerBlockImplementation of Figure 2 and the Multimodal Pathway Transformer with cross modal FFN, plug in and play your FFNRe-Usable and Modular.Combines linear projections from multiple modelsimporttorchfromtorchimportnnfromm2ptimportMPTransformerBlock# Create an instance of the MPTransformerBlock class with the specified parametersmodel=MPTransformerBlock(dim=512,# Dimension of the input and output tensorsdim_head=64,# Dimension of each attention headheads=8,# Number of attention headsdropout=0.1,# Dropout rateff_mult=4,# Multiplier for the dimension of the feed-forward networkoriginal_linear=nn.Linear(512,512),# Linear layer for the original input tensorauxiliar_linear=nn.Linear(512,512),# Linear layer for the auxiliary input tensorffn_original_linear=nn.Linear,# Linear layer for the original input tensor in the feed-forward networkffn_auxiliar_linear=nn.Linear,# Linear layer for the auxiliary input tensor in the feed-forward networkffn_original_last_linear=nn.Linear,# Last linear layer for the original input tensor in the feed-forward networkffn_aux_last_linear=nn.Linear,# Last linear layer for the auxiliary input tensor in the feed-forward network)# Create a 3D tensor with shape B x S x Dx=torch.randn(1,512,512)# Pass the input tensor through the modelout=model(x)# Print the shape of the output tensorprint(out.shape)CrossModalReparameterizationImplementation of the Cross Modal Reparameterization from the paper in Figure 2 and section 3.2It combines the linear methods of different multi-modal models and kinda merges them through addition and a constant value lambda or Cross Modal ScaleModular & Re-usable: Simply plug in your linears from any models!importtorchimporttorch.nnasnnfromtransformersimportBertModel,BertConfig,ViTModel,ViTConfigfromm2ptimportCrossModalReparameterization# Define a simple Transformer model for textclassTextTransformerModel(nn.Module):def__init__(self,bert_model_name='bert-base-uncased'):super(TextTransformerModel,self).__init__()self.bert=BertModel.from_pretrained(bert_model_name)# Assume we're reparameterizing the first linear layer of the classifierself.classifier=nn.Linear(self.bert.config.hidden_size,2)defforward(self,input_ids,attention_mask):outputs=self.bert(input_ids=input_ids,attention_mask=attention_mask)pooled_output=outputs.pooler_outputlogits=self.classifier(pooled_output)returnlogits# Define a simple Transformer model for images (using ViT for example)classImageTransformerModel(nn.Module):def__init__(self,vit_model_name='google/vit-base-patch16-224'):super(ImageTransformerModel,self).__init__()self.vit=ViTModel.from_pretrained(vit_model_name)# Assume we're using the first linear layer of the classifier as the auxiliary layerself.classifier=nn.Linear(self.vit.config.hidden_size,2)defforward(self,pixel_values):outputs=self.vit(pixel_values=pixel_values)pooled_output=outputs.pooler_outputlogits=self.classifier(pooled_output)returnlogits# Example usage# Initialize both modelstext_model=TextTransformerModel()image_model=ImageTransformerModel()# Assume we want to reparameterize the classifier layer of the text model# using the classifier layer of the image modelcross_modal_layer=CrossModalReparameterization(text_model.classifier,image_model.classifier)# Replace the classifier in the text model with the cross-modal layertext_model.classifier=cross_modal_layer# Example input (batch_size, sequence_length)input_ids=torch.randint(0,1000,(8,512))attention_mask=torch.ones(8,512)# Forward pass through the reparameterized modellogits=text_model(input_ids,attention_mask)print(logits)# Train the text model as usual...# After training, merge the parameters for inferencetext_model.classifier.merge_parameters()Citation@misc{zhang2024multimodal,title={Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities},author={Yiyuan Zhang and Xiaohan Ding and Kaixiong Gong and Yixiao Ge and Ying Shan and Xiangyu Yue},year={2024},eprint={2401.14405},archivePrefix={arXiv},primaryClass={cs.CV}}LicenseMIT