Multi-modal VisualBERT can be used for classification task?
Author: karndeepsingh
Pull request: no

Hi,
I have images and descriptions for products and want to train multi-modal by using Image and text embeddings. I just came across VisualBert model and was wondering whether we can use VisualBERT for classification task taking image and text as an input. 

Also, if any other multi-modal algorithm can be recommended apart from VisualBERT to train multi-modal using Image and text for classification task.

thanks

Comments:
Hi,

VisualBERT can probably be used for this, but I'd recommend checking out [ViLT](https://huggingface.co/docs/transformers/model_doc/vilt), which is a very simple extension of ViT (or BERT) for multimodal tasks. The benefit of ViLT over VisualBERT is that one doesn't need to prepare image embeddings, as the ViLT model creates them by itself internally. You can just feed `input_ids` and `pixel_values` to it.

To use ViLT for multimodal classification, you can create a class like so:

```
from transformers import ViltPreTrainedModel, ViltModel
from transformers.modeling_outputs import SequenceClassifierOutput
from torch import nn

class MultimodalClassifier(ViltPreTrainedModel):
      def __init__(self, config):
        super().__init__(config)
        self.config = config
        self.vilt = ViltModel(config)
        self.classifier = nn.Linear(config.hidden_size, config.num_labels)

      def forward(
            self,
            input_ids=None,
            attention_mask=None,
            token_type_ids=None,
            pixel_values=None,
            pixel_mask=None,
            head_mask=None,
            inputs_embeds=None,
            image_embeds=None,
            labels=None,
            output_attentions=None,
            output_hidden_states=None,
            return_dict=True,
        ):
        outputs = self.vilt(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            pixel_values=pixel_values,
            pixel_mask=pixel_mask,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            image_embeds=image_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        pooler_output = outputs.pooler_output if return_dict else outputs[1]

        logits = self.classifier(pooler_output)

        loss = None
        if labels is not None:
            loss_fct = nn.CrossEntropyLoss()
            loss = loss_fct(logits.view(-1, self.config.num_labels), labels.view(-1))

        if not return_dict:
            output = (logits,) + outputs[2:]
            return ((loss,) + output) if loss is not None else output

        return SequenceClassifierOutput(
            loss=loss,
            logits=logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )
```
Creating this model (with a pre-trained base) can then be done as follows:

```
model = MultimodalClassifier.from_pretrained("dandelin/vilt-b32-mlm", num_labels=10)
```
Doing a forward pass on a batch of image+text pairs can be done as follows:

```
from transformers import ViltProcessor
import torch
import requests
from PIL import Image

processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm")

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)
text = "this is an image of two cats"

inputs = processor(image, text, return_tensors="pt")

outputs = model(**inputs, labels=torch.tensor([1]))
```

> Hi,
> 
> VisualBERT can probably be used for this, but I'd recommend checking out [ViLT](https://huggingface.co/docs/transformers/model_doc/vilt), which is a very simple extension of ViT (or BERT) for multimodal tasks. The benefit of ViLT over VisualBERT is that one doesn't need to prepare image embeddings, as the ViLT model creates them by itself internally. You can just feed `input_ids` and `pixel_values` to it.
> 
> To use ViLT for multimodal classification, you can create a class like so:
> 
> ```
> from transformers import ViltPreTrainedModel, ViltModel
> from transformers.modeling_outputs import SequenceClassifierOutput
> from torch import nn
> 
> class MultimodalClassifier(ViltPreTrainedModel):
>       def __init__(self, config):
>         super().__init__(config)
>         self.config = config
>         self.vilt = ViltModel(config)
>         self.classifier = nn.Linear(config.hidden_size, config.num_labels)
> 
>       def forward(
>             self,
>             input_ids=None,
>             attention_mask=None,
>             token_type_ids=None,
>             pixel_values=None,
>             pixel_mask=None,
>             head_mask=None,
>             inputs_embeds=None,
>             image_embeds=None,
>             labels=None,
>             output_attentions=None,
>             output_hidden_states=None,
>             return_dict=True,
>         ):
>         outputs = self.vilt(
>             input_ids,
>             attention_mask=attention_mask,
>             token_type_ids=token_type_ids,
>             pixel_values=pixel_values,
>             pixel_mask=pixel_mask,
>             head_mask=head_mask,
>             inputs_embeds=inputs_embeds,
>             image_embeds=image_embeds,
>             output_attentions=output_attentions,
>             output_hidden_states=output_hidden_states,
>             return_dict=return_dict,
>         )
> 
>         pooler_output = outputs.pooler_output if return_dict else outputs[1]
> 
>         logits = self.classifier(pooler_output)
> 
>         loss = None
>         if labels is not None:
>             loss_fct = nn.CrossEntropyLoss()
>             loss = loss_fct(logits.view(-1, self.config.num_labels), labels.view(-1))
> 
>         if not return_dict:
>             output = (logits,) + outputs[2:]
>             return ((loss,) + output) if loss is not None else output
> 
>         return SequenceClassifierOutput(
>             loss=loss,
>             logits=logits,
>             hidden_states=outputs.hidden_states,
>             attentions=outputs.attentions,
>         )
> ```
> 
> Creating this model (with a pre-trained base) can then be done as follows:
> 
> ```
> model = MultimodalClassifier.from_pretrained("dandelin/vilt-b32-mlm", num_labels=10)
> ```
> 
> Doing a forward pass on a batch of image+text pairs can be done as follows:
> 
> ```
> from transformers import ViltProcessor
> import torch
> import requests
> from PIL import Image
> 
> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm")
> 
> url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
> image = Image.open(requests.get(url, stream=True).raw)
> text = "this is an image of two cats"
> 
> inputs = processor(image, text, return_tensors="pt")
> 
> outputs = model(**inputs, labels=torch.tensor([1]))
> ```

@NielsRogge Thanks for your reply. I will definitely look into ViLT for the IMAGE+ TEXT classification tasks. There are following two things I want to highlight if you can suggest me:
1. I have text in the Spanish language, How I can get the Bert model pre-trained in the Spanish language incorporated here in ViLT implementation?
2. While classifying, i just don't want to classify with single label. Classification is more of an Multi-Label. How I can incorporate this multi-label output in ViLT?

Your suggestion on these two will help me to understand the things properly. 
Thanks again. Waiting for your reply.

> I have text in the Spanish language, How I can get the Bert model pre-trained in the Spanish language incorporated here in ViLT implementation?

ViLT was pre-trained on English text only, unfortunately. In that case, a better alternative might be to just forward the text through a BERT-like model pre-trained on Spanish text (could be multilingual), forward the image through a ViT-like model, and simply concatenate the hidden states of both modalities, which are then fed to a classifier.

> While classifying, i just don't want to classify with single label. Classification is more of an Multi-Label. How I can incorporate this multi-label output in ViLT?

Multi-label classification requires to use the `BCEWithLogitsLoss` as seen [here](https://github.com/huggingface/transformers/blob/1dfa03f12b3748dc7e9c2b5ada40c3401ada23a5/src/transformers/models/bert/modeling_bert.py#L1592) for example. The labels need to be a tensor of shape (batch_size, num_labels), containing the one-hot encoded labels for a batch.

> > I have text in the Spanish language, How I can get the Bert model pre-trained in the Spanish language incorporated here in ViLT implementation?
> 
> ViLT was pre-trained on English text only, unfortunately. In that case, a better alternative might be to just forward the text through a BERT-like model pre-trained on Spanish text (could be multilingual), forward the image through a ViT-like model, and simply concatenate the hidden states of both modalities, which are then fed to a classifier.
> 
> > While classifying, i just don't want to classify with single label. Classification is more of an Multi-Label. How I can incorporate this multi-label output in ViLT?
> 
> Multi-label classification requires to use the `BCEWithLogitsLoss` as seen [here](https://github.com/huggingface/transformers/blob/1dfa03f12b3748dc7e9c2b5ada40c3401ada23a5/src/transformers/models/bert/modeling_bert.py#L1592) for example. The labels need to be a tensor of shape (batch_size, num_labels), containing the one-hot encoded labels for a batch.

Thanks @NielsRogge  for replying.
Can I pass pre-trained tokenizer (pretrained on spanish language) to ViLT processor ?


Also, which model in ViLT shall i use to extract image features in combination with pretrained spanish bert model?

please advise.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.