Pipelines returns inconsistent results when using non-default model
Author: sjgiorgi
Pull request: no
Labels: name: bug

### System Info

Transformers version 4.19.2
Python 3.7.13
Ubuntu 16.04.6 LTS

### Who can help?

@Narsil 

I've noticed that `pipeline` returns inconsistent results, after re-instantiating it, when supplying a non-standard model. See code below.

- What is being returned and why does it change?
- What exactly does `pipeline` do when you give it a non-default model or a model not trained for the specific task? 
- Since it doesn't necessarily make sense to use `bert-base-uncased` for a sentiment analysis task, should pipeline allow this? I don't get a warning or error. Is there a recommended way to tell pipeline to fail if the supplied model doesn't make sense?




### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
>>> from transformers import pipeline
>>> pipe = pipeline("sentiment-analysis", model="bert-base-uncased")
>>> pipe("This restaurant is awesome")
[{'label': 'LABEL_0', 'score': 0.5899267196655273}]

>>> pipe = pipeline("sentiment-analysis", model="bert-base-uncased")
>>> pipe("This restaurant is awesome")
[{'label': 'LABEL_0', 'score': 0.5623320937156677}]

>>> pipe = pipeline("sentiment-analysis", model="bert-base-uncased")
>>> pipe("This restaurant is awesome")
[{'label': 'LABEL_1', 'score': 0.5405012369155884}]

```

### Expected behavior

I would expect pipeline to either fail or give a warning message if given a model not trained for the task. 

Comments:
Hi @sjgiorgi ,

Did you disable the logs somehow ?
When running your code you can see:

```
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.weight', 'cls.predictions.decoder.weight']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

which *does* warn you about uninitialized weights and potential issues.
Maybe you deactivated the warnings ?

Ah, yes, I see those warnings now (I had changed the logging) and apologize.  Thank you.

My concerns are:

- While the warning message says "You should probably TRAIN..." it does not _explicitly_ say "We don't recommend using this model for classification". 
- Results are still being returned, albeit with different labels. It's not clear what these numbers mean or how something like `bert-base-uncased` is being used in a classification setting. Is there documentation on what is happening?

So it's not clear how I should be catching model mismatch given that pipeline is returning something that looks reasonably formatted. Also, blindly adding pipeline to some code could produce some weird results and new users might not fully understand the current warning's implications.

I've handled this on my side, where I check the returned pipeline dictionary results against known keys which _should_ be in the results (e.g., `bert-base-uncased` in the `sentiment-analysis` pipeline returns a label of `LABEL_0` as opposed to `POSITIVE` or `NEGATIVE`):

```
PIPELINE_RESULTS_BY_TASK = {
    "text-classification": ["POSITIVE", "NEGATIVE"], 
    "sentiment-analysis": ["POSITIVE", "NEGATIVE"], 
    "question-answering": ["answer"], 
    "translation": ["translation_text"], 
    "summarization": ["summary_text"], 
    "token-classification": ["entity"], 
    "ner": ["entity"], 
    "text-generation": ["generated_text"], 
}
```

but I'm not sure if this will catch everything. 

@sjgiorgi 

I do agree that it's easy to miss warnings, especially when running setups automatically and serving them for instance, those warnings might not be readily visible to you.

The real culprit here, is that the model architecture you are trying to load is actually very capable of running the pipeline.
But the model weights themselves are missing the layers the architecture is looking for (here it doesn't have the classification head).

Catching the warning would be the best way to be 100% sure it works that way.

Pinging a core maintainer to see if we have other solutions. My personal idea would be to enable a flag to raise a hard error on mismatched weights instead of a warning, and using that flag in pipelines because we really don't want to load from pretrained an incomplete model.
It's a different story in Model.from_pretrained where it's actually a desired feature if you intend to finetune,

@sgugger maybe ?

100% agree on the culprit. People will think "i know bert does classification" and then blindly use this model. Which is what we were doing :) 

Yes, a flag like that would be very useful. Thank you. 

Not really in favor of that flag, as you could have weights not in the checkpoint that are not actually used, and thus still having the pipeline work.

Really ? But we're using `AutoModelForCausalLM` for instance. So extraneous weights can be safely ignored, but missing weights are almost always necessarily used, no ?

Do you have an example of architecture where that fails? 

Thanks for the answer, you're probably right, I just can't find an example from my experience.

What does "work" mean though? In the example above, I'm using `bert-base-uncased` in the sentiment pipeline. As @Narsil pointed out, there is no classification head but a result is returned so in some sense it "works" (maybe we are using different senses of "work"). 

What is that number? How is it being calculated? Why does it change when I re-instantiate the pipeline? Regardless of how to handle all of this, these answers are not clear from the documentation. 

A flag, which could default to the current behavior, would at least allow end users to have some control over this. 

Edit: I see your point @sgugger, and yes we are using different senses of "work". And this explains why the warning message doesn't explicitly say "don't do this" (because it may be the case that it _is_ okay to do this).  Is there a way to distinguish what you had in mind from my example (which doesn't actually work even though it returns well-formatted results)? 

> What is that number? How is it being calculated? Why does it change when I re-instantiate the pipeline? Regardless of how to handle all of this, these answers are not clear from the documentation.

By default the classification is created randomly. Then the correct weights are placed onto your model. Since those weights are missing we just don't place them. That's why outputs change all the time. the head is different all the times.

The problem is that we have *a lot* of architectures and while there shouldn't be any warning in theory if everything has been coded right, I can't guarantee there is not one that shows some warning because some of the internal class variables for the weights that should be ignored in that warning are not properly set (those keys would be tensors that are not set randomly but deterministically like the `position_ids` of BERT). That's why I'm not too much in favor of erroring instead of warning.

Makes sense ! Thanks.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.