Cannot reproduce results from zeroshot demo app
Author: buhrmann
Pull request: no

Using the same text and the same labels I cannot "exactly" reproduce the result from the zeroshot app here https://huggingface.co/zero-shot/, using the "XLM Roberta XNLI" option. See below for my attempt to get the same result.

## Environment info

- `transformers` version: 4.5.1
- Platform: Linux-5.6.0-1055-oem-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.7.1+cpu (False)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help
@joeddav 

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

``` python
from transformers import pipeline

model = "joeddav/xlm-roberta-large-xnli"
classifier = pipeline("zero-shot-classification", model=model, framework="pt")

txt = "There SHOULD be a connection of the GROUND wire to a ground in the breaker box. There also should be a connection of the NEUTRAL wire to a ground in the breaker box. There should be no other place in the building where such a connection occurs (i.e. not in any of the outlet boxes). The NEUTRAL (white) wire is a 'grounding conductor' for the plug, and is NOT safe to touch, while the GROUND (green) wire is a 'protective ground' and carries no current unless some kind of electrical fault has occurred. It's safe to touch the protective ground, but not to touch the grounding conductor (because there is current in the grounding conductor, its outlet-box end will not be at the same ground potential as its breaker-box end)."

template = "This text is about {}"
custom_labels = [
    "politics",
    "politics and guns", 
    "politics and middle east",
    "religion or christianity or atheism",
    "science and cryptography",
    "science and electronics",
    "science and medicine",
    "science and space"
]
res = classifier(txt, candidate_labels=custom_labels, template=template, multi_label=False)
list(zip(res["labels"], res["scores"]))
```

```
[
 ('science and electronics', 0.17324578762054443),
 ('religion or christianity or atheism', 0.15423095226287842),
 ('politics and middle east', 0.12779277563095093),
 ('science and space', 0.1238853707909584),
 ('science and cryptography', 0.12293272465467453),
 ('science and medicine', 0.10926352441310883),
 ('politics', 0.09960934519767761),
 ('politics and guns', 0.08903954923152924)
]
```

## Expected behavior

I was hoping to get something similar to the result from pasting the same text and labels into the app, namely

```
[
 ('science and electronics', 14.4%),
 ('politics', 14%),
 ('science and space', 13.7%),
 ('politics and guns', 13.1%)
 ('politics and middle east', 12.7%),
 ('science and medicine', 11.8%),
 ('science and cryptography', 10.7%),
 ('religion or christianity or atheism', 9.6%)
]
```

Small differences would be expectable I guess, because of potential platform/framework differences etc., but the fact that the "religion or christianity or atheism" category leads to such different results make me wonder if I'm not using the same model as the app, or a different prompt perhaps?

Neither of the two gives particularly great results in this case, but knowing the origin of this difference would be useful for better evaluating the pipeline.

EDIT: I've noticed I've passed the template with the wrong argument name (which didn't fail since `__call__` accepts arbitrary **kwargs), but using the correct one doesn't make the result any more similar to the one from the app.

Comments:
It's subtle but frustratingly important: your hypothesis template needs to have a period at the end. `This text is about {}.` Try that and let me know.

With the period at the end (and the proper argument name `hypothesis_template`), I'm getting

```
[
 ('science and electronics', 0.1435241997241974),
 ('science and cryptography', 0.13319259881973267),
 ('science and space', 0.13153083622455597),
 ('religion or christianity or atheism', 0.12961038947105408),
 ('politics and middle east', 0.12887051701545715),
 ('science and medicine', 0.11486212909221649),
 ('politics', 0.11052283644676208),
 ('politics and guns', 0.10788644850254059)
]
```

The code for the streamlit app is not public, is it? To see what other differences there may be...

(In general, but that's perhaps just the nature of this approach to zeroshot, the results seem quite sensitive to how the hypothesis is formulated. Even seemingly harmless changes may mean that the religious category is suddenly the most probable, which is kind of surprising for the sample text, but yeah that's another story...)

Btw, I'm assuming this message can be ignored

```
Some weights of the model checkpoint at joeddav/xlm-roberta-large-xnli were not used when initializing XLMRobertaForSequenceClassification: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']
- This IS expected if you are initializing XLMRobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing XLMRobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
```

because of how the original model is repurposed?

Yes you can disregard that message safely. And it's not 100% up to date but the code for the demo is public: https://github.com/joeddav/zero-shot-demo

Great, thanks! For some reason I didn't find that repo in your account when I looked. In any case, the only difference I can see is in the creation of the pipeline (the more manual creation using model and tokenizer instances instead of simply the model name). But that doesn't affect the result at all in my tests. So unless the app is somehow using a different version of the model I assume the difference is in how it is executed in different environments.

Hmm so the code in my repo is out of date because we now use the inference API as the backend and it looks like there's a discrepancy between inference API outputs and the pipeline outputs. I'll look into it.

Ok, let me know if I can test anything (small-scale) here to help.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hi, I am dealing too with significant differences between streamlit example and my local testing, is there any update regarding this issue?

@shimonhaf How significant are the changes? It appears that this might be due to the quantization done by the inference API which the demo uses. cc @Narsil 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.