Ability to Finetune ZeoShot Text Classifier on a corpus
Author: m-ali-awan
Pull request: no

### Feature request

I want to do zero-shot text classification for automotive parts, an candidate labels are around 3200. Now this needs to be done on basis of the Description.
Using pretrained Zero-Shot models like Bart-mnli etc, are not giving me good results, as they are not having much context knowledge. It would be great to finetune ZeroShot model on all descriptions corpus, as I think this will improve results a lot. I saw this approach in ULMFit by Fastai, as they train the Language model encoder by training to predict the next word, for the whole corpus. And then they use that encoder as backbone for text classifier, and once that is fine-tuned, results are better.
It can be seen here: 
https://docs.fast.ai/tutorial.text.html

Thanks, 

### Motivation

This way, we can provide rich context, that is required for making classifications/label tagging.

### Your contribution

I am not skilled enough, to contribute on this...


Comments:
@m-ali-awan , I believe they are already implemented in transformers, 
1. `from transformers import AutoModelForMaskedLM` for Masked Language Modelling(like your example) and 
2. `from transformers import AutoModelForCausalLM` for GPT like models. 

> @m-ali-awan , I believe they are already implemented in transformers,
> 
> 1. `from transformers import AutoModelForMaskedLM` for Masked Language Modelling(like your example) and
> 2. `from transformers import AutoModelForCausalLM` for GPT like models.

@susnato 
Thanks for helping me...
But, how can I format my corpus in the format of data required?


@m-ali-awan  For MaskedLM you can just use the pre built class `DataCollatorForLanguageModeling` with mlm=True and mlm_probability=prob(how frequent do you want your tokens to be masked) as data_collator in Trainer class, then you can load your data using huggingface datasets and transformers will take care of all preprocessing in backend. I found a good and brief Kaggle Notebook, about this you can find it [here](https://www.kaggle.com/code/quincyqiang/maskedlm-pretrain-for-deberat-v3-large/notebook).

@susnato 
Thanks a lot. So, now if I want to use BART-MNLI for ZeroShot or Few Shot, I can finetune it on my corpus, in the same way, as described in the mentioned Kaggle notebook. I will do this as MaskedLM, right(or should I treat it as CausalLM)?
Then, using that fine-tuned model, I can do ZeroShot, and FewShot(using around 5-10 labeled examples).


Thanks again.

@m-ali-awan, Yes, but you may need to change the model based on your specific task/dataset, you can search online to see if you find a specific model which was trained for that specific type of task and for that specific type of dataset you are trying to use. You may want to go with variations of BERT for MLM at starting.

Ok thanks @susnato 
So, now I will got for BART-Large-MNLI, and fine tune it as MLM with my custom corpus at first. Then, will try it out for Zero-Shot or Few Shot, and same I can follow for other models...

Thanks


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.