Multiclass image classification with ViT - computer vision
Author: rozeappletree
Pull request: no

Hi, are there building blocks available in the repo to extend current single class to **multilabel classification**? 

It should not be very big of an issue. I guess. 

Comments:
#15978 

And also I would want to know if we can do prediction interpretation by visualizing attention maps or something like that. 

@NielsRogge 

Hi,

You can easily fine-tune ViT (or any other sequence classifier) using the `problem_type` argument.

```
from transformers import ViTForImageClassification

model = ViTForImageClassification.from_pretrained("google/vit-base-patch16--224", problem_type="multi_label_classification")
```
This will make sure the appropriate loss function is used (i.e. `BCEWithLogitsLoss`).

> Hi,
> 
> You can easily fine-tune ViT (or any other sequence classifier) using the `problem_type` argument.
> 
> ```
> from transformers import ViTForImageClassification
> 
> model = ViTForImageClassification.from_pretrained("google/vit-base-patch16--224", problem_type="multi_label_classification")
> ```
> 
> This will make sure the appropriate loss function is used (i.e. `BCEWithLogitsLoss`).

And what about dataset? How do I load it from images folder (specifically for multilabel) 


> And also I would want to know if we can do prediction interpretation by visualizing attention maps or something like that.

This as well please let me know it.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> > Hi,
> > You can easily fine-tune ViT (or any other sequence classifier) using the `problem_type` argument.
> > ```
> > from transformers import ViTForImageClassification
> > 
> > model = ViTForImageClassification.from_pretrained("google/vit-base-patch16--224", problem_type="multi_label_classification")
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > This will make sure the appropriate loss function is used (i.e. `BCEWithLogitsLoss`).
> 
> And what about dataset? How do I load it from images folder (specifically for multilabel)

Hi, im also having trouble with loading the dataset. specifically - how does the model expect the labels for multilabel? did you manage to solve this?

Hi,

For multi-label classification, you need to make sure that you provide `pixel_values` of shape (batch_size, num_channels, height, width) and `labels` of shape (batch_size, num_labels). The latter contain the one-hot encoded labels.

So let's say you have an image that contains a cat and a remote and your labels are cat, remote, dog and house.

In that case, your labels need to be this: `torch.tensor([[1, 1, 0, 0]])`.

Thanks! :)

A tutorial notebook for this is now available here: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SigLIP/Fine_tuning_SigLIP_and_friends_for_multi_label_image_classification.ipynb

> A tutorial notebook for this is now available here: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/SigLIP/Fine_tuning_SigLIP_and_friends_for_multi_label_image_classification.ipynb

Hi,

I've been looking at your notebook, and I've noticed that the loss doesn't always decrease as the epochs progress during training (sometimes it goes up, and other times it goes down). I'm also trying to solve a multi-label classification problem and the same thing happens to me. However, with other published MULTI-CLASS classification problems, this doesn't happen. This is a problem, right?

@cescdovi note that the notebook was made for demo purposes, I'd recommend tweaking the learning rate/warmup schedule/learning rate scheduler etc. a bit more