Automatically cuts input if the pipeline position_ids can't handle it.
Author: Narsil
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Currently the failure is `index out of range in self`.  (it fails in pytorch position_embeddings call).
The current PR aims to simply cut the input to the pipeline if the pipeline can't handle it with a warning to the user.

It feels better than the bad error message. As for triggering an error, it seems correct for the underlying model (it can't handle
the input). But, it seems a bit off to trigger the error for the pipeline as there is no way of knowing how to trim the input
for the user as he does not inputs tokens himself. Automatically cutting with a warning seems a bit better from
a usage standpoint.

This PR also improve a docstring which had a typo.
It also adds a new `PipelineWarning` because there are now 2 instances of such warnings and there should be more in the
future so enabling users to catch those warnings seems like a good idea.



## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@patrickvonplaten
@lhoestq

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

Comments:
I just discovered that roberta does not respect max_embeddings_postion (it adds `padding_idx` to it) so the current fix actually does not really fix it for roberta. If t5 is another special case I fear the current approach is not adapted.



> I just discovered that roberta does not respect max_embeddings_postion (it adds `padding_idx` to it) so the current fix actually does not really fix it for roberta. If t5 is another special case I fear the current approach is not adapted.

I think Roberta does respect `max_embeddings_positions` - why do you think it does not? Roberta is just ugly because `max_position_ids=514 != 512`, but your approach here should work for Roberta no? 

You can check

```python
from transformers import pipeline

pipe = pipeline(task='sentiment-analysis', model='roberta-base-openai-detector')
pipe("Some.....very long text")
```
And it will fail. The reason is:

```python
def create_position_ids_from_input_ids(input_ids, padding_idx):                
    """Replace non-padding symbols with their position numbers. Position numbers begin at
    padding_idx+1. Padding symbols are ignored. This is modified from fairseq's
    `utils.make_positions`.                                                                               
                                                                               
    :param torch.Tensor x:                                                     
    :return torch.Tensor:                                                      
    """                                                                        
    # The series of casts and type-conversions here are carefully balanced to both work with ONNX export and XLA.
    mask = input_ids.ne(padding_idx).int()                                     
    incremental_indices = torch.cumsum(mask, dim=1).type_as(mask) * mask       
    return incremental_indices.long() + padding_idx   
```
As it mentions here, position_embeddings start at `padding_idx+1`. So 2, and finish at 514 (max_embeddings_position), but if we send a sequence of length `514` then the final position_embedding will be `516` (514 + 2) and so out of the embeddings available.

I'll look at T5 to look for a better approach

> ```python
> create_position_ids_from_input_ids
> ```

I see! 

But this also actually looks like a bug in Roberta then. This function does not make much sense in combination with `max_position_embeddings=514`... I think Roberta's `max_position_embeddings` should actually be changed to `512`. @LysandreJik - can you take a look at this as well maybe? 

Regarding T5, as written in the issue I think we should delete `max_position_embeddings`.

I think `T5` is ok, as it seems to use `n_positions` not `max_embedding_positions`, no ?

https://s3.amazonaws.com/models.huggingface.co/bert/t5-base-config.json
https://s3.amazonaws.com/models.huggingface.co/bert/t5-large-config.json

> I think `T5` is ok, as it seems to use `n_positions` not `max_embedding_positions`, no ?
> 
> https://s3.amazonaws.com/models.huggingface.co/bert/t5-base-config.json
> https://s3.amazonaws.com/models.huggingface.co/bert/t5-large-config.json

The config defines `max_position_embeddings` as `n_positions` so it does have a `max_position_embeddings` param

Okay I changed the logic.

Instead of using `truncation='longest_first'`, I am not using it, but I am following the `deprecation_warning` flag to actually be able to re-emit a warning, otherwise it is silent which is a bit damageable I'd say.

There is still an issue, where the small model used in the pipeline tests `"sshleifer/tiny-distilbert-base-uncased-finetuned-sst-2-english"` actually does not define properly a `tokenizer.model_max_length`, but I think it's more a configuration issue than a code issue.

What do you think? 

It's a good idea to be able to re-emit the warnings, however, this dictionary exists so that these warnings are not re-emitted down the line. The point of this is that users only see this warning once and it doesn't flood their stdout.

This means that
```py
>>> from transformers import RobertaTokenizer
>>> tokenizer = RobertaTokenizer.from_pretrained("roberta-base")
>>> tokenizer.encode("Hey how are you" * 1000)
Token indices sequence length is longer than the specified maximum sequence length for this model (4002 > 512). Running this sequence through the model will result in indexing errors
[0, ...]
>>> tokenizer.deprecation_warnings
{'sequence-length-is-longer-than-the-specified-maximum': True}
>>> tokenizer.encode("Hey how are you" * 10)
[0, ...]
>>> tokenizer.deprecation_warnings
{'sequence-length-is-longer-than-the-specified-maximum': True}
```
Checking against it means you will always re-emit the warning:
```py
>>> from transformers import pipeline
... 
... pipe = pipeline(task='sentiment-analysis', model='roberta-base-openai-detector')
... pipe("Some.....very long text" * 1000)
[...]
Token indices sequence length is longer than the specified maximum sequence length for this model (5002 > 512). Running this sequence through the model will result in indexing errors
/home/jik/Workspaces/Python/transformers/src/transformers/pipelines.py:697: PipelineWarning: You input length was too long (5002 > 512) for this model and was truncated.
  PipelineWarning,
>>> pipe("Some.....very long text")
/home/jik/Workspaces/Python/transformers/src/transformers/pipelines.py:697: PipelineWarning: You input length was too long (7 > 512) for this model and was truncated.
  PipelineWarning,
[{'label': 'LABEL_0', 'score': 0.8550598621368408}]
```
Also while you're searching for better solutions, if you can use the `truncation` parameter that would be awesome. If you can't, no big deal, but I'd rather we use our own API if possible.

I am agree with the idea to not spam, the problem is that the current way of doing it is 

```python
                logger.warning(                                                       
                    "Token indices sequence length is longer than the specified maximum sequence length "
                    "for this model ({} > {}). Running this sequence through the model will result in "
                    "indexing errors".format(len(encoded_inputs["input_ids"]), self.model_max_length)
                )  
```

not

```python
warnings.warn(".....", UserWarning)
```

(which btw can be triggered once by setting a warning filter but it's not the purpose here).

The more I think about this issue, the more I think the current behavior might be the correct one for transformers.
- **Silent truncation is worse than raising an exception (ignoring what could be most of the input, is super dangerous for users, as it might output very wrong results).**
- Adding checks (at the model level) within the forward pass is too costly.
- `Re-emitting` a warning is bad if it's not trivial enough to do (don't want too much logic to handle those). Here we would need to capture (logger) before re-emitting (could be another logger or warnings, but it's still a bit much of work) 
- At the pipeline level, we don't want to dive too deeply into model internals (like model_max_length and other configs), I feel at most config that is shared by *all* or *almost all* models (like vocab_size)

I'm in favor of closing this PR in the end.

Yes, the reason we use `logging` instead of `warnings` is to make use of the centralized logging system.
I agree with all your points, and I believe this is something that could be patched by allowing the users to pass kwargs to the tokenizer/model, as is something that should be enabled in pipelines v2. 

Closing then, thank you for experimenting!