Repository: huggingface/transformers

Issue:
Mismatch between tokenizer and model in pipeline
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.6.1
- Platform: Linux-4.18.0-25-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Nope
- Using distributed or parallel set-up in script?: Nope

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
@LysandreJik
## Information

Model I am using (Bert, XLNet ...): Any models with `pipeline`.

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce
```python
In [14]: p = pipeline("sentiment-analysis", tokenizer='cardiffnlp/twitter-roberta-base-sentiment')  # only tokenizer is provided

In [15]: p.tokenizer  # roberta as provided
Out[15]: PreTrainedTokenizerFast(name_or_path='cardiffnlp/twitter-roberta-base-sentiment', vocab_size=50265, model_max_len=1000000000000000019884624838656, is_fast=True, padding_side='right', special_tokens={'bos_token': '
', 'eos_token': '
', 'unk_token': '<unk>', 'sep_token': '
', 'pad_token': '<pad>', 'cls_token': '
', 'mask_token': AddedToken("<mask>", rstrip=False, lstrip=True, single_word=False, normalized=False)})

In [21]: p.model.config.model_type  # falls back to hard coded default model
Out[21]: 'distilbert'

In [22]: p("What a lovely day")  # does not work
Out[22]: [{'label': 'NEGATIVE', 'score': 0.8819105625152588}]
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
<!-- A clear and concise description of what you would expect to happen. -->
The tokenizer and model should be compatible regardless of how arguments to pipeline are given.
I think the if statements in the [`pipeline` function](https://github.com/huggingface/transformers/blob/2d1d92181a0f739b6817a74401c51862d28bb409/src/transformers/pipelines/__init__.py#L265) should be something like below to handle all the cases.
```python
if model is not None and tokenizer is not None:
    # case 1: use default     
elif model is None and tokenizer is None:
    # case 2: model should follow tokenizer
elif model is not None and tokenizer is None:
    # case 3: tokenizer should follow model
elif model is None and tokenizer is not None:
    # case 4: maybe assert if the two are compatible?
```
In addition, although the [current code](https://github.com/huggingface/transformers/blob/2d1d92181a0f739b6817a74401c51862d28bb409/src/transformers/pipelines/__init__.py#L426) complains that we cannot infer tokenizer when model is given as a `PreTrainedModel` (one scenario under case 3), I think it is possible through `AutoTokenizer.from_pretrained(model.config._name_or_path)` as `_name_or_path` is `'bert-base-cased', for example.

Let me know how you think! I would be happy to submit a PR if consensus is reached :)


Discussion:
It's not possible to just mix and match tokenizers with models. The `sentiment-analysis` pipeline uses the `DistilBertForSequenceClassification` model by default. However, you are providing it with a RoBERTa-based tokenizer. DistilBERT does make use of WordPiece tokenization, whereas RoBERTa-like models make use of a BPE (Byte-Pair Encoding) tokenizer.

What's a good use case of only providing a tokenizer to a pipeline, but not a model?
Thanks @NielsRogge for a quick reply.

I totally agree with you that no one would want to mix and match, and it is not even possible in many cases as you pointed out.

And that's why I suggested for a model to follow the compatible tokenizer, or vice versa.

Or at least, I think raising an error or printing a warning message would be nice for users who could only provide tokenizer by mistake, since for now it leads to a unwanted results without any warning. (On a personal note, a warning message from transformers library was a huge time saver for me :) )
I think we could indeed raise an error/better warning if only a tokenizer is provided. When the model is provided, the tokenizer is selected automatically from that ID, which I agree is a bit weird as it doesn't work the other way.

I think erroring out when the `tokenizer` is especially specified but not the model would be nice to prevent unseen errors from happening. Is there a use-case I'm not seeing @Narsil?
I don't think so, even then, specifying forcibly model + tokenizer should work (and crash later if not compatible) as it becomes a user responsability IMO.

So raising error when specifying tokenizer and NOT model seems the best here.
I see that many agree to raise an error.  So maybe add something like below at [the beginning](https://github.com/huggingface/transformers/blob/2d1d92181a0f739b6817a74401c51862d28bb409/src/transformers/pipelines/__init__.py#L382) of the pipeline function?
```python
if model is None and tokenizer is not None:
  raise Exception(
    "Impossible to instantiate a pipeline with tokenizer specified but not the model."
    "Please provide a PreTrainedModel class or a path/identifier to a pretrained model when providing tokenizer."
  )
```

Plus, how about allowing selecting tokenizer from the model that is provided as PreTrainedModel, which is [not supported for the moment](https://github.com/huggingface/transformers/blob/2d1d92181a0f739b6817a74401c51862d28bb409/src/transformers/pipelines/__init__.py#L426)? Perhaps with something like this?
```python
# else:
#     # Impossible to guess what is the right tokenizer here
#     raise Exception(
#         "Impossible to guess which tokenizer to use. "
#         "Please provide a PreTrainedTokenizer class or a path/identifier to a pretrained tokenizer."
#     )
elif isinstance(model, PreTrainedModel):
  tokenizer = model.config._name_or_path
```
(I found that `model.config._name_or_path` is set [inside `PreTrainedModel.from_pretrained`](https://github.com/huggingface/transformers/blob/2d1d92181a0f739b6817a74401c51862d28bb409/src/transformers/modeling_utils.py#L1269), which I believe every model passes through)

I agree with your first proposal! Do you want to open a PR?

For your second proposal, how would that differ from the line above:
```
if isinstance(model_name, str):
    tokenizer = model_name
```
?
checking `model_name` if it's str does not cover the following case where a model is passed as a `PreTrainedModel`.
```python
In [4]: model = AutoModel.from_pretrained(""cardiffnlp/twitter-roberta-base-sentiment")
In [5]: p = pipeline(model=model, task="sentiment-analysis")
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-7-57dfcbcb4261> in <module>
----> 1 p = pipeline(model=model, task="sentiment-analysis")

~/anaconda3/lib/python3.8/site-packages/transformers/pipelines/__init__.py in pipeline(task, model, config, tokenizer, feature_extractor, framework, revision, use_fast, use_auth_token, model_kwargs, **kwargs)
    437             else:
    438                 # Impossible to guess what is the right tokenizer here
--> 439                 raise Exception(
    440                     "Impossible to guess which tokenizer to use. "
    441                     "Please provide a PreTrainedTokenizer class or a path/identifier to a pretrained tokenizer."

Exception: Impossible to guess which tokenizer to use. Please provide a PreTrainedTokenizer class or a path/identifier to a pretrained tokenizer.
```

I think the second proposal makes the pipeline guess the correct tokenizer when `model` is provided as `PreTrainedModel` as well.

I will open a PR as soon as we settle on the second proposal :)
For your second proposal, I actually think the current error is pretty good and point users to a valid solution.

Automatically deriving the tokenizer from model.config._name_or_path seems a bit "too much magic" to me :

- Either, you just want magic and pass strings, or you want to be in full control and pass real objects.
- I don't think we can rely on  `_name_or_path` to be the actual desired valid path for the tokenizer. The actual object might have been modified before the call, making the path obsolete (you can also have a directory without any tokenizer, imaging AutoModel.from_pretrained(....), model.save_pretrained('mydirectory'))
- The fact that the variable is marked as "private" leads me to think that pipeline shouldn't rely on it.

Just a personal opinion. In any case the failing error should be as descriptive as possible.

Because we now have audio and image pipelines, the feature_extractor also needs the same logic regardless of the chosen solution IMO.



Resolving patch:
diff --git a/src/transformers/pipelines/__init__.py b/src/transformers/pipelines/__init__.py
--- a/src/transformers/pipelines/__init__.py
+++ b/src/transformers/pipelines/__init__.py
@@ -379,6 +379,18 @@ def pipeline(
         >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
         >>> pipeline('ner', model=model, tokenizer=tokenizer)
     """
+    if model is None and tokenizer is not None:
+        raise RuntimeError(
+            "Impossible to instantiate a pipeline with tokenizer specified but not the model "
+            "as the provided tokenizer may not be compatible with the default model. "
+            "Please provide a PreTrainedModel class or a path/identifier to a pretrained model when providing tokenizer."
+        )
+    if model is None and feature_extractor is not None:
+        raise RuntimeError(
+            "Impossible to instantiate a pipeline with feature_extractor specified but not the model "
+            "as the provided feature_extractor may not be compatible with the default model. "
+            "Please provide a PreTrainedModel class or a path/identifier to a pretrained model when providing feature_extractor."
+        )
 
     # Retrieve the task
     targeted_task, task_options = check_task(task)


Test patch: