[Proposal] Adding infinite generation as an option to generate
Author: Narsil
Pull request: yes

# What does this PR do?

`generate` is limited in a lot of models, where we can't generate
more tokens than the limit of a said model (`seq_length`,
`max_position_embeddings`).

It corresponds to a reality of how models where trained, but advanced
usage like inference might rely on using the models to generate longer
output than this. It is also quite inconvenient to hit that barrier when
using pipelines (where inputs are text and not tokens).

So the main goal is to make this behavior non-default but opted
in by users as they should understand what is happening and the
limits linked to this behavior.

The following proposal is to enable (model per model) infinite
generation. It *might* be doable generally, however `past` seems
to be model dependant so it could be harder to do in general.

The implementation here simply clips any left values if somehow
the `input_ids` (or `past`) is larger than what the model can cope with.
We also propose to enable that by default for `text-generation`
models (`text2text-generation` might also make use of it).

Happy to hear your thoughts on this:

1- Should we enable that kind of feature ?
2- Is optional, with enabled by default for pipelines correct ?
3- Can we enable for every models instead of on a model-per-model basis ?


<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@patrickvonplaten @LysandreJik @patil-suraj

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

Comments:
I get what you are saying, it's a valid concern.

`position ids` tend to be very well correlated with their siblings, so it shouldn't matter (so much) for the first few iterations out of bounds.
It will matter more at full fledged `seq_length` drift, but I figure it's the callers responsibility to understand those caveats (hence the need for actual explicit decision required within `generate`). 
I ran some tests and the drift is much more substantial than what I expected for the full `seq_length`. the 20 topk tokens will share between 0 and 10 between non drifted and drifted versions. [gist](https://gist.github.com/Narsil/468af7fe59eaf1e20eb03ec0a4c9d249)

We also need to take into account that some models do not have position embeddings, or other position schemes (sinusoidal), that might change the perspective on this ?

Disabling the cache might be an alternative, I guess the caller should know what are the tradeoffs he wants to make.

Again it is just that, for `pipelines` it is very hard to reason in number of tokens, and you can be hit by walls concerning number of tokens, without any recourse.

An example would be summarization, If I try to summarize some text too large to fit my model, I receive error 3051 tokens > 1024 tokens. It's fine, but now, how much of the string should I cut to actually get a summary ? It's impossible to know. cascading summaries is an option. It has some drawbacks but maybe that's still what I am looking for ?

What I'm suggesting is coping mechanisms within `pipelines` than can be opted in to circumvent the issues mentioned above.
`aggregation_strategy` is a good example of such a thing that was added in `token-classification`. It's a way to cope with incorrect/undesirable outputs from models to produce better end results to users, even if it's not perfectly representative of the model.



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.