Tokenizer encode to have an option to overflow from left
Author: misrasaurabh1
Pull request: no
Labels: name: wontfix

name: High-Level feature

name: Core: Tokenization

# 🚀 Feature request

Current tokenizer encode variants ( encode, batch_encode, batch_encode_plus) handle longer sequences than max_length by overflowing tokens from the right hand side and thus restricting the length to max_length. This feature request is to allow an option for the tokenizer encode methods to overflow tokens from the left hand side as well.


## Motivation

For problems dealing with dialog, if one were to train an intent classification or next sentence prediction model and the dialog was longer than max_length, one would like to throw away the tokens from the beginning of the conversation as they are less relevant than the more recent messages.

This motivates the need for a encoder that works well with dialog data where more recent tokens are more valuable.

## Your contribution

I could change the function `truncate_sequences` by adding a new truncation_strategy option that will truncate from left. But want to get feedback from the Huggingface team about this proposal.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@misrasaurabh1 What solution do you use now for this encoding dialog data problem?

I use something like
`self.tokenizer.encode(input)[-self.block_size:]`
This throws a warning for length overflow so I deactivate it with logging. 
Also one has to make attention masks separately as some models require this.

Indeed, we should add an option to truncate on the left!
cc @n1t0 for our sprint of September.

perhaps add a truncation_side to https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer to be consistent with padding_side.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@thomwolf @n1t0 Any plan for this? I just saw this because of the bot.

I think I can do this, seems like all the logic is here. 
https://github.com/huggingface/transformers/blob/07708793f20ec3a949ccab32cc4fe0c7272dcc4c/src/transformers/tokenization_utils_base.py#L2766

But how about fast 🤗 Tokenizers? Will I need to also change the rust code?

And I noticed something that might be a bug, and can be improved:

https://github.com/huggingface/transformers/blob/07708793f20ec3a949ccab32cc4fe0c7272dcc4c/src/transformers/tokenization_utils_base.py#L2816-L2831

Here it loops `num_tokens_to_remove` times to decide how many tokens needs to be truncated for each sequence, which can be calculated without looping.

And in case `stride` is not 0, it seems to return up to `stride`*`num_tokens_to_remove` extra tokens to `overflowing_tokens`.
https://github.com/huggingface/transformers/blob/07708793f20ec3a949ccab32cc4fe0c7272dcc4c/src/transformers/tokenization_utils_base.py#L2801-L2803
Also it seems weird to me that `overflowing_tokens` will be mixed with tokens from `ids` and `pair_ids`. Perhaps it should be a tuple of list if `TruncationStrategy` is `longest_first`.

Note to self: `overflowing_tokens` is used in squad to construct another pair if the doc is too long. `stride` is also used in squad. I can't find other use of `overflowing_tokens`.

https://github.com/huggingface/transformers/blob/969859d5f67c7106de4d1098c4891c9b03694bbe/src/transformers/data/processors/squad.py#L154-L216

One feedback about what's happening with this facility of left truncation being not available - its harder to use the datasets library and we have to do python Hackery which reduces the benefits of using the datasets library in the first place.

I recently needed to do exactly this, but ran into this issue so I had to manually truncate the text. Simply doing `encoded_tensor[-max_length:]` would also truncate samples that are less than `max_length` since they are padded to the right.  

Here's the approach I used instead:
```python
def encode_right_truncated(tokenizer, text, padding='max_length', max_length=512, add_special_tokens=True):
    tokenized = tokenizer.tokenize(text, padding=padding, max_length=max_length, add_special_tokens=add_special_tokens)
    
    if not add_special_tokens:
        truncated = tokenized[-max_length:]
    else:
        truncated = tokenized[0:1] + tokenized[-(max_length-1):]
    
    ids = tokenizer.convert_tokens_to_ids(truncated)
    
    return ids
```

Hope this helps future people finding this from Google/DDG

For anyone arriving here from search, note that this is now possible by setting [`truncation_side`](https://huggingface.co/docs/transformers/main/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.truncation_side) 
```python
# specify when initializing the tokenizer,
tokenizer = AutoTokenizer.from_pretrained(..., truncation_side = "left")
# or modify an already-initialized tokenizer, like
tokenizer.truncation_side = "right"
```
see https://github.com/huggingface/transformers/pull/12913