Cannot replicate the scores from the pipeline
Author: DanieleBarreca
Pull request: no

I am trying to build a custom function that replicates the transfomers' pipeline in order to optimize runtimes for the roberta model, like in [this section of the documentation](https://huggingface.co/joeddav/xlm-roberta-large-xnli#with-manual-pytorch). 
The blueprint to write my custom function is the  [ZeroShotClassificationPipeline](https://github.com/huggingface/transformers/blob/v4.17.0/src/transformers/pipelines/zero_shot_classification.py). 
Even though I managed to run the model and get similar results, I cannot fully replicate the scores coming from the pipeline. 

Below the example:

## 1. ZeroShotClassificationPipeline
```
classifier_cuda = pipeline("zero-shot-classification",
                      model=os.path.abspath('../models/xlm-roberta-large-xnli'),
                      device=torch.cuda.current_device()
                     )

classifier_cuda("Mi sono divertito a teatro ieri", ["money", "entertainment", "food", "rent"],hypothesis="This example is related to {}")
```

The ouput scores related to this example are the following:
```
{'sequence': 'Mi sono divertito a teatro ieri',
 'labels': ['entertainment', 'rent', 'money', 'food'],
 'scores': [0.9842655062675476,
  0.010558217763900757,
  0.003464417764917016,
  0.0017118172254413366]
}
```

## 2. Custom function
```
device = torch.device('cuda:0')

tokenizer = AutoTokenizer.from_pretrained(os.path.abspath('../models/xlm-roberta-large-xnli'))
nli_model = AutoModelForSequenceClassification.from_pretrained(os.path.abspath('../models/xlm-roberta-large-xnli')).to(device)

sequence = "Mi sono divertito a teatro ieri"
hypothesis_template = "This example is related to {}"
labels = ["money", "entertainment", "food", "rent"]

sequence_pairs = []
sequence_pairs.extend([[sequence, hypothesis_template.format(label)] for label in labels])

inputs=tokenizer(
    sequence_pairs,
    add_special_tokens=True,
    padding=True,
    return_tensors='pt',
    truncation='only_first'
)

for key,value in inputs.items():
    inputs[key]=value.to(device)

model_out = nli_model(**inputs)

entailments = model_out['logits'].cpu().detach().numpy()[:,nli_model.config.label2id['entailment']]

entailments = entailments.reshape((-1,len(labels)))
entailments_logits=np.exp(entailments)
result=(entailments_logits/entailments_logits.sum(-1, keepdims=True))
print(labels)
print(list(result[0]))
```

Results:
```
['money', 'entertainment', 'food', 'rent']
[0.0014175308, 0.99700063, 0.00071506965, 0.00086674973]
```


Comments:
cc @Narsil 

Hi @DanieleBarreca ,

Can you try with `hypothesis_template` as a variable, I think that's the issue.

@LysandreJik , We're very tolerant on arguments, what do you think about raising errors on pipelines that do not accept `*kwargs` when the passed argument is not understood (would prevent silent errors like this one)

@Narsil 
Thank you very much for your support. Your suggestion is not super-clear to me unfortunately, can you help me to understand in which function call should I pass the hypothesis_template as a varibale (i.e. to the tokenizer, to the model, ...) or how should I modify my code? 

Sorry about that, here is the modified example to get the same scores. 
```python
classifier_cuda = pipeline("zero-shot-classification",
                      model=os.path.abspath('../models/xlm-roberta-large-xnli'),
                      device=torch.cuda.current_device()
                     )

classifier_cuda(
  "Mi sono divertito a teatro ieri",
  ["money", "entertainment", "food", "rent"],
  # THIS LINE IS DIFFERENT
  hypothesis_template="This example is related to {}"
)
```
Basically you `hypothesis` argument was ignored and it used the default instead


@Narsil Thank you very much! It worked correctly and I can replicate the pipeline scores.