Truncation + max_length not working for GPT2TokenizerFast
Author: teetone
Pull request: no

### System Info

```shell
- `transformers` version: 4.13.0
- Platform: Linux-4.15.0-29-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- PyTorch version (GPU?): 1.10.2+cu102 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
```


### Who can help?

@patil-suraj, @patrickvonplaten, @LysandreJik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Run the following Python code:

```Python

from typing import List

from transformers import GPT2TokenizerFast


DESIRED_TOKEN_LENGTH: int = 1949
TEXT = 'Title: Hilltop Hoods\n\nBackground: Hilltop Hoods are an Australian hip hop group that formed in 1994 in Blackwood, Adelaide, South Australia. The group was founded by Suffa (Matthew David Lambert) and MC Pressure (Daniel Howe Smith), who were joined by DJ Debris (Barry John M. Francis) after fellow founder, DJ Next (Ben John Hare), left in 1999. The group released its first extended play, Back\n\nSection: 2007-2009: The Hard Road Restrung and State of the Art\nPassage: Two of Hilltop Hoods\' founders first met in 1987 when MC Suffa (aka Matthew David Lambert) and MC Pressure (Daniel Howe Smith) attended Blackwood High School in Eden Hills - a suburb of Adelaide. In 1991 they joined up with DJ Next (Ben John Hare) through a mutual friend and formed an Australian hip hop group. Their name was supplied by fellow local MC Flak (from Cross Bred Mongrels) - the suburb of Blackwood is known by locals as the Hilltop. The band\'s influences include American hip hop artists: Notorious B.I.G., KRS-One, Gang Starr, Wu-Tang Clan and Public Enemy. At live shows Next was the group\'s DJ, for recording he contributed audio engineering and all the scratching/turntablism on their early works. He regularly competed in the local DMC World DJ Championships (DMC) tournaments, winning the South Australian DMC championships multiple times. Hilltop Hoods recorded a demo, Highlanders, which was released on cassette tape only. As well as Pressure and Suffa on vocals, the group included MC Summit aka DJ Sum-1, but he did not appear on later Hilltop Hoods work. The group\'s first official release, in 1997, was a vinyl-only, seven-track extended play, Back Once Again. Production was handled by DJ Debris (Barry John M Francis), turntablism and audio engineering by Next, vocals by Pressure and Suffa. The third track, "Shades of Grey", features Debris with a verse, and was co-written by Francis, Hare, Lambert and Smith. Fifth track, "Mankind Must Suffa" also features a guest verse from Quromystix (aka Quro, Andrew Michael Bradley) - a member of Finger Lickin\' Good and later the Fuglemen. "Mankind Must Suffa" is credited to Lambert, Smith, Francis and Bradley. Back Once Again is out of print and unavailable for retail purchase. The group\'s debut studio album, A Matter of Time, was released in 1999 on CD only. As with Back Once Again, it is now unavailable for retail purchase. All scratching/turntablism is performed by Next, a track, "Let Me Show You", has no vocals - solely showcasing his turntable skills. American MC Bukue One (Tion Torrence) appears for a guest verse on "Deaf Can Hear". The track is credited to Lambert, Smith, Francis, Hare and Torrence. The album was released independently but with financial assistance from Arts SA - the band were inspired, in 2005, to set up their own Hilltop Hoods Initiative, to help local artists. After the album appeared, Next left the group and moved to Melbourne. In 2004 he moved to London. In 1999 Debris, who was also a member of the Cross Bred Mongrels, replaced Next and became the Hilltop Hoods\' full-time DJ. Hilltop Hoods founded the Certified Wise Crew - a hip hop collaborative - with local groups Terra Firma, Cross Bred Mongrels and After Hours. Certified Wise Crew has since expanded to include MCs Trauma, Blockade, Kolaps, Flea, with Vents and Funkoars joining in later years. Hilltop Hoods received two nominations for the Hip Hop Act of the Year Award at the Australian Dance Music Awards and again at the 3D World Music Awards in 2001 and 2002. In 2001 the group\'s second album, Left Foot, Right Foot, was released with Lambert, Francis and M. Veraquth producing. On 22 September 2003, Hilltop Hoods released their third album, The Calling, which became a commercial breakthrough. In an interview after the release of their fourth album, Suffa revealed that The Calling was recorded on his mother\'s computer and the simplicity of their \'studio\' is the reason why some of the music on the album is in monaural (\'mono\') sound. The Calling entered the ARIA Albums Chart in March 2004 and reached No. 53 before exiting the top 100 in September of the same year. By December 2006 it was certified platinum for shipment of 70,000 units, becoming the first Australian hip hop album to achieve platinum status. In March 2012, it re-entered the chart and peaked at No. 50 - eight-and-a-half years after its first release. It featured two singles, "The Nosebleed Section" and "Dumb Enough", which were listed in the Triple J Hottest 100, 2003. "The Nosebleed Section" was ranked No. 17 in the Triple J Hottest 100 of All Time in 2009. Hilltop Hoods\' chart and commercial success was a turning point in the Australian Hip Hop scene because it demonstrated widespread support for the genre that reached beyond an underground fan base. On 1 April 2006, the group followed with their fourth album, The Hard Road, which peaked at number one. It was the first Australian hip hop album to do so. It was certified gold within a week of being released. Its lead single, "Clown Prince", reached the top 30 on the related ARIA Singles Chart. It featured guest verses from New York rapper, Omni, and British MCs, Mystro and Braintax. The Hilltop Hoods received the inaugural Australian Independent Record (AIR) Award for Independent Artist of the Year and Best Performing Independent Album for The Hard Road in 2006. The track, "The Blue Blooded", is a collaboration with Australian MCs: Funkoars, Hau from Koolism, Mortar, Vents, Drapht, Muph & Plutonic, Pegz and Robby Balboa. On 27 April of the same year, Hilltop Hoods performed at the Bass in the Grass music festival in Darwin alongside fellow hip hop group, The Herd. That same day they issued a second single, the title track from the album. Its video includes fellow members from the Certified Wise Crew - Cross Bred Mongrels, Terrafirma and Funkoars. Following the success of The Hard Road Tour in early 2006, the Hilltop Hoods began their second national tour for the year, The Stopping All Stations Tour, which visited more regional areas of Australia as well as the capital cities. They were supported by Koolism and Mystro. Late that year, Hilltop Hoods released their third single from the album, "What a Great Night". The video shows the group at a club with camera shots panning up and down to reveal a new location. It used special effects and is one of the most expensive video clips for an Australian hip hop group, mirroring the group\'s rise in success and popularity. Also late in the year the band won the J Award for best album of the year from Triple J. They performed the Homebake Festival and Falls Festival before the end of the year. The Hard Road received the AIR Award for Best Independent Hip Hop/Urban Release in 2007. On 12 May 2007, Hilltop Hoods released their next album The Hard Road: Restrung which is a remix of their previous studio album, The Hard Road, featuring the Adelaide Symphony Orchestra and Okwerdz. It peaked at No. 8 on the ARIA Albums Chart. Like its predecessor The Hard Road, it took out "Best Urban Release" at the ARIA Awards of 2007, with the group going back-to-back in the category. The lead single from the album "Recapturing the Vibe Restrung", its video clip was on high rotation on rage & jtv. That year the group performed at the Southbound Festival (WA), The Great Escape at Newington Armory over Easter, and embarked on a UK tour with a Sydney-based string quartet. They finished the year by headlining the Pyramid Rock Festival on Victoria\'s Phillip Island over New Year\'s Eve 2007. In 2008 they performed at the Big Day Out festivals, at Glastonbury Festival and Islington Academy in London. In December their DVD, The City of Light, was released and was nominated as \'Best Music DVD\' at the 2008 ARIA Awards. Hilltop Hoods left their longtime home of Obese Records to start their own label, Golden Era Records, to release their future material. In November 2008 Pressure announced on Triple J\'s breakfast program that the next studio album, State of the Art, would be recorded with session musicians: "We realised with this one after doing Restrung and having an orchestra that we were a bit less limited. So we\'re going to have some session musos come in on this one and stuff like that". The album was released on 12 June, with the lead single, "Chase That Feeling", issued as a digital download on 8 May, and featured a return guest appearance by a quartet from the Adelaide Symphony Orchestra. The album debuted at number one on the albums chart while "Chase That Feeling" peaked at No. 8 on the related singles chart. By 2010 State of the Art was certified 2x platinum for shipment of 140,000 units. In early 2009 the Hilltop Hoods performed at the Groovin the Moo festival in Townsville, Maitland and Bendigo. They also performed at Triple J\'s One Night Stand in Sale, Victoria on 30 May, and at Fat as Butter festival in Newcastle on 25 October where they played several of the tracks from the album. To promote its release the band started a national tour starting on 18 July and performed at most major cities including state capitals. The second national tour that year followed on 11 November with support provided by Vents.\n\nQuestion: What is significant about this time?\nAnswer: On 1 April 2006, the group followed with their fourth album, The Hard Road,\n\nQuestion: How did this album do?\nAnswer: which peaked at number one. It was the first Australian hip hop album to do so.\n\nQuestion: Are there any other interesting aspects about this article?\nAnswer:'

tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")
tokens: List[int] = tokenizer.encode(TEXT, truncation=True, max_length=DESIRED_TOKEN_LENGTH)
assert len(tokens) == DESIRED_TOKEN_LENGTH  # 1949, no problem here

result: str = tokenizer.decode(tokens)
print(len(tokenizer.tokenize(result)))  # 1950 when it should be 1949
print(len(tokenizer.encode(result)))    # 1950 when it should be 1949
assert len(tokenizer.tokenize(result)) == DESIRED_TOKEN_LENGTH  # Fails here!
```

It should fail at the last assertion:

```
Traceback (most recent call last):
  File "/Users/tonyhlee/research/mercury/benchmarking/gpt2_tokenizer_bug.py", line 16, in <module>
    assert len(tokenizer.tokenize(result)) == DESIRED_TOKEN_LENGTH  # Fails here!
AssertionError
```

### Expected behavior

Since I passed in `truncation=True` and `max_length=1949` to `encode`, I would expect the resulting text to be 1949 tokens long after decoding. It's 1950 tokens long instead.



Comments:
Also gently pinging @SaulLu @mishig25 @thomasw21 here

Hi @teetone,

Thank you for your detailed outcome. While investigating, I noticed that the portion of text that produces this behaviour is `"the simplicity of their 'studio' is the reason why"`

```python
from transformers import AutoTokenizer


DESIRED_TOKEN_LENGTH = 1949
TEXT="their 'studio'"

tokenizer = AutoTokenizer.from_pretrained("gpt2")
encoding_1 = tokenizer.encode(TEXT, truncation=True, max_length=DESIRED_TOKEN_LENGTH)
print(f"Encoding 1st time is of length {len(encoding_1)} and corresponds to {tokenizer.convert_ids_to_tokens(encoding_1)}")

decoded_encoding_1 = tokenizer.decode(encoding_1)
# 
encoding_2 = tokenizer.encode(decoded_encoding_1)
print(f"Encoding 2nd time is of length {len(encoding_2)} and corresponds to {tokenizer.convert_ids_to_tokens(encoding_2)}")
print(f"Decoded sequence of ids is \"{decoded_encoding_1}\"")
# Encoding 1st time is of length 5 and corresponds to ['their', "Ġ'", 'stud', 'io', "'"]
# Encoding 2nd time is of length 6 and corresponds to ['their', "'s", 't', 'ud', 'io', "'"]
# Decoded sequence of ids is "their'studio'"
```
The reason is that by default the `clean_up_tokenization_spaces` argument is set to true and has the effect of removing the space between `their` and `'studio'`. By specifying this argument to False you get:
```python
from transformers import AutoTokenizer


DESIRED_TOKEN_LENGTH = 1949
TEXT="their 'studio'"

tokenizer = AutoTokenizer.from_pretrained("gpt2")
encoding_1 = tokenizer.encode(TEXT, truncation=True, max_length=DESIRED_TOKEN_LENGTH)
print(f"Encoding 1st time is of length {len(encoding_1)} and corresponds to {tokenizer.convert_ids_to_tokens(encoding_1)}")

decoded_encoding_1 = tokenizer.decode(encoding_1, clean_up_tokenization_spaces=False)
encoding_2 = tokenizer.encode(decoded_encoding_1)
print(f"Encoding 2nd time is of length {len(encoding_2)} and corresponds to {tokenizer.convert_ids_to_tokens(encoding_2)}")
print(f"Decoded sequence of ids \"{decoded_encoding_1}\"")
# Encoding 1st time is of length 5 and corresponds to ['their', "Ġ'", 'stud', 'io', "'"]
# Encoding 2nd time is of length 5 and corresponds to ['their', "Ġ'", 'stud', 'io', "'"]
# Decoded sequence of ids "their 'studio'"
```
I hope this answers your problem!


> 

@SaulLu, thanks for your help! I'm using this `encode` + `decode` logic to truncate text to fit in a given context window. Is it safe to say that if I want to truncate while preserving the original text, I should pass in `clean_up_tokenization_spaces=False` when calling `decode`?

Generally it is not promised that 1-1 matching is possible. But in the particular case of GPT-2 (without added tokens or special tokens present in the sentence to be tokenized) I think it should work with `clean_up_tokenization_spaces=False` in the `decode` method!

> Generally it is not promised that 1-1 matching is possible. But in the particular case of GPT-2 (without added tokens or special tokens present in the sentence to be tokenized) I think it should work with `clean_up_tokenization_spaces=False` in the `decode` method!

That makes sense. Thank you again! I will close this issue as resolved.