Input Packing
Author: Sanger2000
Pull request: no

### Feature request

Sequence packing when tokenizing inputs.

Most modern large language models pack together multiple sequences to saturate their large context windows. Otherwise, they risk wasted computation with excess padding. For example T5, GPT3, and PALM all implemented input packing. 

"During training we always train on sequences of the full nctx = 2048 token context window, packing multiple
documents into a single sequence when documents are shorter than 2048, in order to increase computational efficiency.
Sequences with multiple documents are not masked in any special way but instead documents within a sequence
are delimited with a special end of text token, giving the language model the information necessary to infer that
context separated by the end of text token is unrelated. This allows for efficient training without need for any special
sequence-specific masking." [1]

"We use a maximum sequence length of 512 and a batch size of 128 sequences. Whenever possible, we “pack” multiple sequences into each entry of the batch..." [2]

I would suggest a change to tokenizers. When tokenizing multiple sequences, sufficiently small inputs are automatically packed together with a special delimiter token separating them. The simplest approach would be a greedy method, where inputs under something like 70% of the context window are added to a queue. Or when truncating longer inputs, the remaining chunk is added to the queue.

When the queue grows to be larger than the size of the window, the first $n_{window}$ tokens are flushed and added as another input.

A more aggressive strategy would be something like this - https://arxiv.org/pdf/2107.02027.pdf

In addition, it would be useful to generate and supply optional masks to the model that prevent models from attending to different sequences.

[1] - Brown, Tom, et al. "Language models are few-shot learners." Advances in neural information processing systems 33 (2020): 1877-1901.
[2] - Raffel, Colin, et al. "Exploring the limits of transfer learning with a unified text-to-text transformer." J. Mach. Learn. Res. 21.140 (2020): 11.

### Motivation

I have been dealing with a dataset with a very high variance in its sequence lengths. I implemented something like this for myself to pack inputs together and thought it could be quite useful as a general feature.

### Your contribution

I could probably make the greedy packer and support the custom masks if I have time in the next few weeks.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@Sanger2000 this is cool - surprised you didn't get much interest. Would you be willing to expand on your approach here so that others can pick it up even if it doesn't get incorporated into the library?

Specifically:

1. what is the delimiter token you used
2. how did you modify the masks/outputs/model to handle the packed inputs?

If you have any pointers to existing code or other details, please share them here.

#6661 is another issue that asks about packing