Add support for T5 models in Zero Shot Classification pipeline
Author: osainz59
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR adds support for T5 like models to be used with the `ZeroShotClassificationWithT5` pipeline. It works exactly like `ZeroShotClassification` pipeline.

I decided to keep them separated in two different classes, and not implement one over the other in order to keep the code more simple. But, maybe can be interesting to put both together as they share the same input and solve the same task. What do you think?

I still have not written the documentation. I want to know the comunity response to the question asked above first.

The performed tests are equal to the `ZeroShotClassificationPipelineTests` without the `_test_entailment_id` because is no longer necessary.

Any suggestion is welcome.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Hello @osainz59! The pipelines are model agnostic, they shouldn't be model-specific with a `With{MODEL}`. I would instead improve the existing zero-shot classification pipeline to add support for T5. Would you be willing to try that out?

> 
> 
> Hello @osainz59! The pipelines are model agnostic, they shouldn't be model-specific with a `With{MODEL}`. I would instead improve the existing zero-shot classification pipeline to add support for T5. Would you be willing to try that out?

Sure!

Changed! What do you think @LysandreJik ?

> 
> 
> Hi @osainz59 ,
> 
> First of all, thanks for this contribution, it's very interesting to be doing zero-shot in a different way.
> 
> Couple of points/questions:
> 
>     * Do you have a benchmark of results or something we can compare with current pipeline version to showcase it has some merit ? (Being similar in performance would still be helpful IMO)
> 

Not yet, but I am interested on how can affect the knowledge adquired from other GLUE and SuperGLUE tasks at the time of building a zero-shot system. This was my motivation to adapt the ZeroShotClassificaitonPipeline to work with T5 like models. I do not have any benchmark yet but it could be interesting to compare with [Yin et al. (2019)](https://aclanthology.org/D19-1404.pdf) (the standard NLI based zero-shot). 

>     * Anything t5 specific needs to be removed. Pipelines don't want to deal with anything model specific (as much as reasonably possible). Your code could work basically on any generating model (maybe limiting to encoder-decoder for simplicity) so we would need to update to adapt
> 

Okay. Initially, I implemented it with only the T5 model in mind, but you are right, this could/should work with any other generative model.

>     * Pipelines are undergoing a pretty major change so might be more interesting to wait for it to be merged.

Okay :)



If you want to focus on the benchmark, that would be helpful, I started the migration part and probably could integrate generative models as well when we're closer to merging. The better the benchmark, the more incentive to get it merge into the library :)

You could very well use this branch to get your benchmark running too btw with heavily customized code and we can figure out later how to make it more general.

I will focus on the benchmark then. Once I get some results I will post them here.

I have run some evaluations over 4 different datasets using task agnostic and task specific `hypothesis_template`s to compare 2 standard NLI models (`roberta-large-mnli` and `facebook/bart-large-mnli`) with the T5 NLI model (`t5-large`).

I ran the experiments without any previous development, let's say, `hypothesis_template` or `candidate_labels` exploration, to simulate a real zero-shot situation where no development data is available. This may derive on a suboptimal performance of the models. The task agnostic `hypothesis_template` used is the default template: `"The example is {}."`. The `t5-large` model is evaluated using the MNLI and RTE tasks prefixes.

## Task definition
### AG News ("ag_news")
Topic classification task over a collection of news articles. 
*Candidate labels:* **World**, **Sports**, **Business** and **Science & Technology**.
*Task specific template:* `"Topic: {}."`

### Yelp ("yelp_review_full")
Rating estimation (from 1 to 5 stars).
*Candidate labels:* **1 star**, **2 star**, **3 star**, **4 star** and **5 star**.
*Task specific template:* `"Review rating: {}."`

### Yelp polarity ("yelp_polarity")
Sentiment classification of restaurants reviews.
*Candidate labels:* **Negative** and **Positive**.
*Task specific template:* `"Polarity: {}."`

### Yahoo ("yahoo_answers_topics")
Topic classification of questions and answers.
*Candidate labels:* **Society & Culture**, **Science & Mathematics**, **Health**, **Education & Reference**, **Computers & Internet**, **Sports**, **Business & Finance**, **Entertainment & Music**, **Family & Relationships** and **Politics & Government**.
*Task specific template:* `"Topic: {}."`


## Results

| Model | Task specific template     | AG News | Yelp | Yelp polarity | Yahoo | Avg |
|------:|:----------------------:|:----------:|:-----:|:-------------:|:------:|:-----:|
| roberta-large-mnli |               | 42.17 | 40.56 | 84.13 | 28.64 | 48.87 |
| roberta-large-mnli | &#x2611;      | **73.43** | 39.15 | 91.61 | 51.59 | 63.94 |
| facebook/bart-large-mnli |         | 66.52 | 33.50 | 90.56 | 48.03 | 59.65 |
| facebook/bart-large-mnli | &#x2611; | 72.64 | 37.52 | 86.52 | 54.15 | 62.70 |
| t5-large (mnli prefix) |           | 53.78 | 31.81 | 90.09 | 47.72 | 55.85 | 
| t5-large (mnli prefix) | &#x2611;   | 51.5  | 34.30 | 92.78 | 45.13 | 55.92 |
| t5-large (rte prefix) |            | 65.06 | 39.33 | 82.52 | **54.40** | 60.32 |
| t5-large (rte prefix) | &#x2611;    | 64.42 | **45.41** | **93.75** | 54.12 | **64.42** |

In terms of perfomance the `t5-large` is pretty similar with the rest I think. Something that can be concluded from this results is that some `hypothesis_template`s works better with some models than others, like for instance on the Yelp dataset almost every model improve the performance when a specific template is used except for `roberta-large-mnli`. There are other cases too. Another thing to emphasize is that the RTE prefix seems to be a better option for the T5 than the MNLI prefix.



@osainz59 

Thank you very much for this ! This is indeed very interesting.
Good to know it works so well out-of-the box.  

The work for new pipelines has been done here: https://github.com/Narsil/transformers/tree/iterable_pipelines
It does not include T5 zero-shot for now, but given the magnitude of the change, it will have to wait probably to implement T5 later (lots of testing chnages first to make sure nothing is breaking, then big refactor PR, then T5-zero-shot)

As for the design of this, I think we should have 2 subclasses of ZeroShot being chosen based on the model (ForGeneration vs ForSequenceClassification) so that the code is nice and clean and we don't have any T5 specific code. For the default hypothesis_template, keeping the current one as the default for all seems like a good enough because we cannot infer on what type of dataset pipelines are going to be used.

Okay!

I have some doubts about the design, I am agree to keep them separated (as my initial proposal, but without T5 specific code), but re-naming the pipeline as ZeroShot (ForGeneration) might be confused with GPT-3 like prompting based zero-shot (https://twitter.com/BigscienceW/status/1429787756063043588?s=20). Also, this last method might be also implemented in the future, so we should make it clear that this pipeline still uses a model fine-tuned on a NLI task, even if it uses a generative model. 

The large PR for pipeline refactoring is here:

https://github.com/huggingface/transformers/pull/13308

Will implement the t5 for zero shot after it is merged (or not). It's a pretty large PR with large impact, so it might take a while.

@osainz59 Do you mind sharing your benchmarking code ? I would like to run some tests on the zero-shot pipeline too and it would be great to use it as a starting point.


Sure! @Narsil  https://drive.google.com/file/d/1SXFEBo24tG1COv2Vm1roB4r5PbsoMzsA/view?usp=sharing
I uploaded it to Drive, maybe I should have shared it as a commit in this PR. 


It's fine that way, maybe as a gist might be a little more future proof or inline (it's not that long), but that's good enough, thanks ! 


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.