Zero Shot Classification Pipeline gives poor results locally than online demo
Author: nerdimite
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 4.0.1
- Platform: Colab
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0 Yes
- Tensorflow version (GPU?): No
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->
@julien-c @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): facebook/bart-large-mnli

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. I have a small dataset of 26 examples and I want to classify them into 2 classes. I first ran all the examples in the [online demo](https://huggingface.co/zero-shot/) and got around 80% accuracy.
2. Then I ran the code on Colab and got only 53% accuracy which I think is just a random answer between the labels.
3. I am aware of the fact that this issue has been opened before and resolved but it isn't working for me. ([This is the previous issue](https://github.com/huggingface/transformers/issues/8122))

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
```python
from transformers import pipeline
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-mnli")
model = AutoModelForSequenceClassification.from_pretrained("facebook/bart-large-mnli")

classifier = pipeline(task='zero-shot-classification', model=model, tokenizer=tokenizer)

hypothesis_template = 'This text is about {}.'
labels = ['Single Patient', 'Multiple Patient']

def predict(sequence, labels, hypothesis_template):
    results = classifier(sequence, labels,
                         hypothesis_template=hypothesis_template)
    pred_idx = np.array(results['scores']).argmax()
    pred_cls = labels[pred_idx]
    return pred_idx, pred_cls

def evaluate(dataset, labels, hypothesis_template):
    n_correct = 0
    for sequence, label in tqdm(dataset.values):
        _, pred = predict(sequence, labels, hypothesis_template)
        n_correct += (pred == label)    
    acc = n_correct / len(dataset)
    print('Accuracy:', acc)

patients = pd.read_csv('patient_classification.csv')
evaluate(patients, labels, hypothesis_template)
```

While loading the model I get this warning message.
```
Some weights of the model checkpoint at facebook/bart-large-mnli were not used when initializing BartForSequenceClassification: ['model.encoder.version', 'model.decoder.version']
- This IS expected if you are initializing BartForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BartForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
The results of the online demo and my local code (Colab) are supposed to be the same.

Comments:
Maybe @joeddav has an idea!

The pipeline output is sorted from highest to lowest scores, so in your code `pred_idx` will always be `0` and `pred_cls` will always be `"Single Patient"`. Instead you want,

```python
pred_cls = results['labels'][0]
pred_idx = labels.index(pred_cls)
```

Oh lol, I didn't know it was that simple xD. Thanks @joeddav that increased the accuracy to 73% (though less than online demo) which is good enough. Thank you so much!