Zero shot classification pipeline
Author: joeddav
Pull request: yes

This PR adds a pipeline for zero-shot classification using pre-trained NLI models as demonstrated in our [zero-shot topic classification demo](https://huggingface.co/zero-shot/) and [blog post](https://joeddav.github.io/blog/2020/05/29/ZSL.html). 

Addresses #5756, where @clmnt requested zero-shot classification in the inference API. However, it should be noted that this model has a max sequence size of `1024`, so long documents would be truncated to this length when classifying.

The pipeline takes in collection of sequences and labels. The labels are converted into a hypothesis, e.g. `vulgar` ➡ `this example is vulgar.` Each sequence and each candidate label must be paired and passed through the model, so the total number of forward passes is `num_labels * num_sequences`.

#### Usage

The pipeline can be initialized using the `pipeline` factory:

```python
from transformers import pipeline
nlp = pipeline("zero-shot-classification")
```
Then any combination of sequences and candidate labels can be passed.
```python
sequence_to_classify = "Who are you voting for in 2020?"
candidate_labels = ["Europe", "public health", "politics"]
nlp(sequence_to_classify, candidate_labels)
>>> {'sequence': 'Who are you voting for in 2020?',
     'labels': ['politics', 'Europe', 'public health'],
     'scores': [0.9676316380500793, 0.019536184147000313, 0.012832209467887878]}
```
When more than one label is passed, we assume that there is only one true label and that the others are false so that the output probabilities add up to 1. This can be changed by passing `multi_class=True`:
```python
sequence_to_classify = "Who are you voting for in 2020?"
candidate_labels = ["Europe", "public health", "politics", "elections"]
nlp(sequence_to_classify, candidate_labels, multi_class=True)
>>> {'sequence': 'Who are you voting for in 2020?',
     'labels': ['politics', 'elections', 'Europe', 'public health'],
     'scores': [0.9720695614814758,
      0.967610776424408,
      0.060417089611291885,
      0.03248738870024681]}
```
The single-label case is likely to be more reliable, however, since the guarantee of only one true label provides strong signal to the model which is very useful in the zero-shot setting.

By default, labels are turned into NLI-format hypotheses with the template `This example is {label}.`. You can change this by with the `hypothesis_template` argument, but the default template seems to work well in most settings I've experimented with.

A couple more examples:
```python
reviews = [
     "I didn't care for this film, but the ending was o.k.",
     "There were some weak moments, but the movie was pretty good overall"
]
nlp(reviews, ["positive", "negative"])
>>> [{'sequence': "I didn't care for this film, but the ending was o.k.",
      'labels': ['negative', 'positive'],
      'scores': [0.9887893199920654, 0.011210653930902481]},
     {'sequence': 'There were some weak moments, but the movie was pretty good overall',
      'labels': ['positive', 'negative'],
      'scores': [0.6071907877922058, 0.3928091824054718]}]
```
```python
reviews = [
     "I didn't care for this film, but the ending was o.k.",
     "There were some weak moments, but the movie was pretty good overall"
]
hypothesis_template = 'The sentiment of this review is {}.'
nlp(reviews, ["positive", "negative"], hypothesis_template=hypothesis_template)
>>> [{'sequence': "I didn't care for this film, but the ending was o.k.",
      'labels': ['negative', 'positive'],
      'scores': [0.9774571061134338, 0.022542938590049744]},
     {'sequence': 'There were some weak moments, but the movie was pretty good overall',
      'labels': ['positive', 'negative'],
      'scores': [0.9787198305130005, 0.021280216053128242]}]
```
```python
nlp("I am a bit discouraged by my grades.", ["sad", "happy", "angry"])
>>> {'sequence': 'I am a bit discouraged by my grades.',
     'labels': ['sad', 'angry', 'happy'],
     'scores': [0.9630885124206543, 0.0311590563505888, 0.005752446595579386]}
```

Comments:
> The UI for inputting a list of potential labels + possibly a hypothesis template is non trivial

I think for purposes of the early inference API, customizing the hypothesis template is not all that important. I added support for providing labels as a list of comma-delimited strings instead of a list.

> one model is currently linked to one (and one only) Pipeline type in the inference API.

Would it be possible to have one model, e.g. `bart-large-mnli`, linked to the zero shot pipeline and have the others remain linked to text classification?

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5760?src=pr&el=h1) Report
> Merging [#5760](https://codecov.io/gh/huggingface/transformers/pull/5760?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/8ab565a4be5a7fd96b19ef88d474037ef31f27e5&el=desc) will **increase** coverage by `0.72%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5760/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5760?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5760      +/-   ##
==========================================
+ Coverage   77.32%   78.05%   +0.72%     
==========================================
  Files         146      146              
  Lines       26047    26089      +42     
==========================================
+ Hits        20141    20363     +222     
+ Misses       5906     5726     -180     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5760?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/5760/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `78.47% <100.00%> (+1.50%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5760/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9vcGVuYWkucHk=) | `20.27% <0.00%> (-74.92%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/5760/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `64.90% <0.00%> (-33.90%)` | :arrow_down: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5760/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `81.02% <0.00%> (-1.29%)` | :arrow_down: |
| [src/transformers/tokenization\_utils\_base.py](https://codecov.io/gh/huggingface/transformers/pull/5760/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHNfYmFzZS5weQ==) | `92.44% <0.00%> (-1.17%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5760/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `85.96% <0.00%> (-0.26%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/5760/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9yb2JlcnRhLnB5) | `93.36% <0.00%> (+49.37%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_mobilebert.py](https://codecov.io/gh/huggingface/transformers/pull/5760/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9tb2JpbGViZXJ0LnB5) | `96.77% <0.00%> (+73.38%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5760?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5760?src=pr&el=footer). Last update [8ab565a...afcf86a](https://codecov.io/gh/huggingface/transformers/pull/5760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


> Would it be possible to have one model, e.g. `bart-large-mnli`, linked to the zero shot pipeline and have the others remain linked to text classification?

Yes we can override manually

So I think I found a possible bug in the calculation of the probabilities. Consider this example:
```python
sequences = ["Who are you voting for in 2020?"]
candidate_labels = ["politics", "public health", "economics", "elections"]

classifier(sequences[0], candidate_labels)
```
which gives the output:
```
{'labels': ['politics', 'elections', 'economics', 'public health'],
 'scores': [0.5225354433059692,
  0.4626988470554352,
  0.007836099714040756,
  0.006929598283022642],
 'sequence': 'Who are you voting for in 2020?'}
```

I was able to replicate the probability calculations by doing the following:
```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-mnli")
model = AutoModelForSequenceClassification.from_pretrained("facebook/bart-large-mnli")

sequences = ["Who are you voting for in 2020?"]
candidate_labels = ["politics", "public health", "economics", "elections"]
template = "This example is {}."
x = tokenizer(sequences*len(candidate_labels), [template.format(label) for label in candidate_labels], return_tensors="pt", padding=True)
with torch.no_grad():
    y = model(**x)[0]
    probs = F.softmax(y[:,-1], dim=0) # Think this is wrong

print(probs)
# tensor([0.5225, 0.0069, 0.0078, 0.4627])
```

So I think the probability calculation is wrong. `y` is of shape `(4, 3)` and the softmax should be over the 3 not the the 4 dimensions, since it is over contradiction, neutral, entailment. So what I am suggesting is that the probability calculation should be the following instead:
```python
probs2 = F.softmax(y, dim=-1)
probs2 = probs2[:,-1] / sum(probs2[:,-1])
print(probs2)
# tensor([0.4977, 0.0016, 0.0049, 0.4958])
```

It's especially important to do it this way since there is no guarantee that the exponential of `y` will sum to one since they are logits.


Hey @sachinruk, thanks for the comment. This isn't a bug though, it's just the way we've chosen to use the outputs. When `multi_class=False`, we ignore the contradiction and neutral logits and just do a softmax over the entailment logits. This does guarantee you will sum to 1. Your snippet is an alternative way of interpreting the model outputs to get probabilities that sum to 1, but I don't see a reason to think it is more correct.

I have included this in my local Jupyter notebook:

!pip install git+https://github.com/huggingface/transformers.git --user

nlp = pipeline("zero-shot-classification") gives the following error.
KeyError: "Unknown task zero-shot-classification, available tasks are ['feature-extraction', 'sentiment-analysis', 'ner', 'question-answering', 'fill-mask', 'summarization', 'translation_en_to_fr', 'translation_en_to_de', 'translation_en_to_ro', 'text-generation']"

What could be the missing steps? Thanks.

@SophiaLeigh you probably just had another version of transformers installed already so the pip install didn’t do anything. Try just passing`--upgrade` with the pip install.

EDIT: 

**pip install transformers** delivers a version of pipelines.py that is not the one found in the current master

**pip install pip install git+https://github.com/huggingface/transformers**  delivers the correct version obviously. 

I dont know anything about the inner workings of pip but I have the current version _pip                       20.2.2_  installed.


Same issue as @SophiaLeigh  with 

_transformers              3.0.2_                 

---------------------------------------------------------------------------
```
KeyError                                  Traceback (most recent call last)
<ipython-input-6-1f0825594ce1> in <module>
----> 1 classifier = pipeline("zero-shot-classification")

~/.conda/envs/zero-shot/lib/python3.7/site-packages/transformers/pipelines.py in pipeline(task, model, config, tokenizer, framework, **kwargs)
   1819     # Retrieve the task
   1820     if task not in SUPPORTED_TASKS:
-> 1821         raise KeyError("Unknown task {}, available tasks are {}".format(task, list(SUPPORTED_TASKS.keys())))
   1822 
   1823     framework = framework or get_framework(model)

KeyError: "Unknown task zero-shot-classification, available tasks are ['feature-extraction', 'sentiment-analysis', 'ner', 'question-answering', 'fill-mask', 'summarization', 'translation_en_to_fr', 'translation_en_to_de', 'translation_en_to_ro', 'text-generation']"
    
```




Also huggingface shows either outdated or wrong info  here: 

https://huggingface.co/transformers/main_classes/pipelines.html?highlight=pipelines

and here:

https://huggingface.co/transformers/_modules/transformers/pipelines.html#pipeline

`Args:
        task (:obj:`str`):
            The task defining which pipeline will be returned. Currently accepted tasks are:

            - "feature-extraction": will return a :class:`~transformers.FeatureExtractionPipeline`
            - "sentiment-analysis": will return a :class:`~transformers.TextClassificationPipeline`
            - "ner": will return a :class:`~transformers.TokenClassificationPipeline`
            - "question-answering": will return a :class:`~transformers.QuestionAnsweringPipeline`
            - "fill-mask": will return a :class:`~transformers.FillMaskPipeline`
            - "summarization": will return a :class:`~transformers.SummarizationPipeline`
            - "translation_xx_to_yy": will return a :class:`~transformers.TranslationPipeline`
            - "text-generation": will return a :class:`~transformers.TextGenerationPipeline``

While I can clearly see it`s here :)

https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines.py

I installed via pip install transformers and also via pip install git+https/thisrepository and ~~both versions~~ have the correct pipelines.py file that has all the implementations for zero-shot- Now I am really confused. 

the error persists though I have included --upgrade 

`pip install git+https://github.com/huggingface/transformers --upgrade
!pip install git+https://github.com/huggingface/transformers --upgrade
Collecting git+https://github.com/huggingface/transformers
  Cloning https://github.com/huggingface/transformers to c:\users\sophia\appdata\local\temp\pip-req-build-6_7vqcz3
Requirement already satisfied, skipping upgrade: numpy in c:\users\sophia\anaconda3\lib\site-packages (from transformers==3.0.2) (1.19.1)
Requirement already satisfied, skipping upgrade: tokenizers==0.8.1.rc2 in c:\users\sophia\anaconda3\lib\site-packages (from transformers==3.0.2) (0.8.1rc2)
Requirement already satisfied, skipping upgrade: packaging in c:\users\sophia\anaconda3\lib\site-packages (from transformers==3.0.2) (20.4)
Requirement already satisfied, skipping upgrade: filelock in c:\users\sophia\anaconda3\lib\site-packages (from transformers==3.0.2) (3.0.12)
Requirement already satisfied, skipping upgrade: requests in c:\users\sophia\anaconda3\lib\site-packages (from transformers==3.0.2) (2.24.0)
Requirement already satisfied, skipping upgrade: tqdm>=4.27 in c:\users\sophia\anaconda3\lib\site-packages (from transformers==3.0.2) (4.48.2)
Requirement already satisfied, skipping upgrade: regex!=2019.12.17 in c:\users\sophia\anaconda3\lib\site-packages (from transformers==3.0.2) (2020.7.14)
Requirement already satisfied, skipping upgrade: sentencepiece!=0.1.92 in c:\users\sophia\anaconda3\lib\site-packages (from transformers==3.0.2) (0.1.91)
Requirement already satisfied, skipping upgrade: sacremoses in c:\users\sophia\anaconda3\lib\site-packages (from transformers==3.0.2) (0.0.43)
Requirement already satisfied, skipping upgrade: six in c:\users\sophia\anaconda3\lib\site-packages (from packaging->transformers==3.0.2) (1.15.0)
Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in c:\users\sophia\anaconda3\lib\site-packages (from packaging->transformers==3.0.2) (2.4.7)
Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\users\sophia\anaconda3\lib\site-packages (from requests->transformers==3.0.2) (1.25.9)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in c:\users\sophia\anaconda3\lib\site-packages (from requests->transformers==3.0.2) (2020.6.20)
Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in c:\users\sophia\anaconda3\lib\site-packages (from requests->transformers==3.0.2) (3.0.4)
Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in c:\users\sophia\anaconda3\lib\site-packages (from requests->transformers==3.0.2) (2.10)
Requirement already satisfied, skipping upgrade: joblib in c:\users\sophia\anaconda3\lib\site-packages (from sacremoses->transformers==3.0.2) (0.16.0)
Requirement already satisfied, skipping upgrade: click in c:\users\sophia\anaconda3\lib\site-packages (from sacremoses->transformers==3.0.2) (7.1.2)
Building wheels for collected packages: transformers
  Building wheel for transformers (setup.py): started
  Building wheel for transformers (setup.py): finished with status 'done'
  Created wheel for transformers: filename=transformers-3.0.2-py3-none-any.whl size=873793 sha256=c363c4ea6e4ec438b88c6fafb162915d24a2c149dd4210111713e78496940af2
  Stored in directory: C:\Users\sophia\AppData\Local\Temp\pip-ephem-wheel-cache-ck6yr140\wheels\35\2e\a7\d819e3310040329f0f47e57c9e3e7a7338aa5e74c49acfe522
Successfully built transformers
Installing collected packages: transformers
  Attempting uninstall: transformers
    Found existing installation: transformers 3.0.2
    Uninstalling transformers-3.0.2:
      Successfully uninstalled transformers-3.0.2
Successfully installed transformers-3.0.2`

@SophiaLei did you restart your kernel after upgrading?
@Tabernakel In the top left corner of the docs, click on `v3.0.2` and switch to `master`.

Feel free to open a topic on https://discuss.huggingface.co with any other questions.

It works now. Thank you.

So the online demo has two different models MNLI and MLNI + Yahoo Answers. I know the second one is Bart with a classification head trained on MNLI and then further fine-tuned on Yahoo Answers topic classification. Is there a specific scenario where MNLI + yahoo answers outperform just MNLI in the zero-shot classification task? 

@avinregmi For practical usage the base MNLI model is typically going to do better. The Yahoo Answers model will do better on Yahoo Answers, which seems unhelpful unless you recognize that was only fine-tuned on 5 labels out of the 10 in the corpus. So if you have a big labeled dataset but only covering a subset of the labels you want to be able classify into, fine-tuning an MNLI model as I did with Yahoo Answers will likely boost your performance. Otherwise, stick with the base MNLI model.

Is there a way to persist Zero shot classification pipeline and use it for deploying in production?

Thanks!

> Is there a way to persist Zero shot classification pipeline and use it for deploying in production?

@mariyamiteva You have two options:

1. Use the pipeline directly via our inference API and let us (Hugging Face engineers) take care of all the production serving for you. Check out [the documentation for our inference API](https://api-inference.huggingface.co/docs/python/html/index.html) and reach out to api-enterprise@huggingface.co to discuss further if you're interested. cc @jeffboudier 

2. Use [this distillation script](https://github.com/huggingface/transformers/tree/master/examples/research_projects/zero-shot-distillation) I wrote with some unlabeled data to distill the zero-shot pipeline to a more efficient sequence classification model, which can then be shared and deployed like any model.

Thanks @joeddav - and @mariyamiteva you can do both: 2) Distill the model to efficient sentence classification and then 1) upload it as a private model on Hugging Face to serve it via our hosted inference API.

Thanks @joeddav and @jeffboudier for the prompt feedback!

I am currently trying to run `distill_classifier.py`. `roberta-large-mnli` has been used as a teacher:

    python distill_classifier.py

    --data_file ..\model\output\unlabeled.txt 
    --class_names_file ..\model\output\class_names.txt
    --teacher_name_or_path roberta-large-mnli
    --multi_label 1
    --output_dir ..\model\output\distilled

My `unlabeled.txt` has a single line text, e.g. 

![image](https://user-images.githubusercontent.com/48296792/118127261-221a1100-b402-11eb-8d64-43f931c5b52c.png)


My `class_names.txt` takes the following form:

![image](https://user-images.githubusercontent.com/48296792/118127065-da938500-b401-11eb-802c-8c97ff099061.png)





Here is the snippet of the error I get:

     File "distill_classifier.py", line 338, in <module>
    main()
      File "distill_classifier.py", line 328, in main
        trainer.train()
      File "C:\Users\Maria\anaconda3\lib\site-packages\transformers\trainer.py", line 1222, in train
        tr_loss += self.training_step(model, inputs)
      File "C:\Users\Maria\anaconda3\lib\site-packages\transformers\trainer.py", line 1617, in training_step
        loss = self.compute_loss(model, inputs)
      File "distill_classifier.py", line 119, in compute_loss
        target_p = inputs["labels"]
      File "C:\Users\Maria\anaconda3\lib\site-packages\transformers\tokenization_utils_base.py", line 230, in __getitem__
        return self.data[item]
    KeyError: 'labels'

Unfortunately, I was not able to deal with the error above. Could you please help? Thanks!

@mariyamiteva I'm not positive this is the issue, but you might need to end your `unlabeled.txt` with a newline. Also, you don't need the `'` single quotes around your text or class names.

Yes, it was not the issue. I performed the proposed changes and upgraded the version to 4.7.0.dev0. 

The error occurred is not the same, but similar:

     Traceback (most recent call last):
      File "distill_classifier.py", line 338, in <module>
        main()
      File "distill_classifier.py", line 328, in main
        trainer.train()
      File "C:\Users\Maria\anaconda3\lib\site-packages\transformers\trainer.py", line 1261, in train
        tr_loss += self.training_step(model, inputs)
      File "C:\Users\Maria\anaconda3\lib\site-packages\transformers\trainer.py", line 1734, in training_step
        loss = self.compute_loss(model, inputs)
      File "distill_classifier.py", line 119, in compute_loss
        target_p = inputs["labels"]
    KeyError: 'labels'

@mariyamiteva I'm not sure this is this issue, but do you have tokenizers installed? Try `pip install tokenizers` and let me know what that gives you.

@joeddav Could the following transformer model used for zero-shot classification be optimized in terms of model inference through ONNX - ‘joeddav/xlm-roberta-large-xnli’?

Thanks!
M.

@joeddav What if anyone wants to fine-tune the zero-shot classifier for a specific domain dataset. Is there any code or GitHub repo that may help us to train the zero-shot classifier?