Truncating the prefix of a sequence rather than the suffix
Author: yuvalkirstain
Pull request: no
Labels: name: WIP

# 🚀 Feature request

Hi, tokenizers get `truncation` as an argument. When set to `True` the tokenizer will truncate the suffix of a sequence so it does not surpass the specified `max_length`. I'd like to have a functionality that truncates the prefix of the sequence, so the model will see the suffix of the sequence.

## Motivation

In many applications (e.g. Dialog, and QA) the most important part of the sequence is the suffix (e.g. the question after the context, or the last response of the dialog).

## Your contribution

Perhaps I'll submit a PR, but it might take me some time as I'm close to some deadlines of mine :(


Comments:
There's a `TruncationStrategy` called `"only_first"` that implements this. See [this](https://github.com/huggingface/transformers/blob/ba15fe7995a02357ecea6e7024918f6915564c36/src/transformers/tokenization_utils_base.py#L125) for all possible truncation strategies. 

@NielsRogge Perhaps I miss something, but it doesn't seem to implement this functionality. The documentation says that it truncates the first *sequence* and not the first *tokens* of the sequence, right?

```:obj:`'only_first'`: Truncate to a maximum length specified with the argument :obj:`max_length` or to
                  the maximum acceptable input length for the model if that argument is not provided. This will only
                  truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided.```

I'm not sure why you mean by truncating the prefix of a sequence.

For question answering, one typically provides `[CLS] question [SEP] context [SEP]` to the model (so question being the first sequence, context being the second sequence). People are usually interested in either truncating the tokens of the question or the context.

What do you mean by prefix/suffix?

We had a misunderstanding. If I use T5/GPT for question answering, the model will receive as input a **single** sequence. This input might look as follows: 
```Background: <first sentence in the context> ... <last sentence in the context>\nQuestion: <question>\nAnswer:```.
Now, if I truncate the **suffix** of the input it might end up as:
```Background: <first sentence in the context> ... <last sentence in the context>```. 
Thus, I will prefer to truncate the **prefix** of the input so the model will get 
```<third sentence in the context>... <last sentence in the context>\nQuestion: <question>\nAnswer:```.

Are my intentions clear now?

If we think about implementation, perhaps we can add flags that signal which part of the sequence we wish to truncate - prefix, or suffix?

Additionally, in many tasks even BERT will receive a single input. A good example might be intent detection of an ongoing dialog. I think that it is unnatural to divide a dialog that is made out of multiple turns into two sequences. However, for intent detection, the most important part of the sequence might be the last turns. Thus, cutting the start of the sequence (prefix) rather than the end (suffix) is probably preferable. 

Ok I get it. Perhaps this could be defined as an additional argument called `truncation_side` (similar to `padding_side`), which can be either "left" or "right".  

`Padding_side` is already implemented as can be seen [here](https://github.com/huggingface/transformers/blob/ba15fe7995a02357ecea6e7024918f6915564c36/src/transformers/tokenization_utils_base.py#L1387) (as one of the attributes when initializing a tokenizer).

perfect! Thanks for simplifying it :)

I think if a `truncation_side` is defined, then it should be used in the `truncate_sequences` function defined [here](https://github.com/huggingface/transformers/blob/ba15fe7995a02357ecea6e7024918f6915564c36/src/transformers/tokenization_utils_base.py#L2925). It could then be used by all different truncation strategies.

Let me implement this :)

Thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.