How to detokenize a BertTokenizer output? 
Author: al-yakubovich
Pull request: no
Labels: name: Discussion

name: wontfix

name: Core: Tokenization

For example, let's tokenize a sentece "why isn't Alex' text tokenizing":

    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
    tokens = tokenizer.tokenize("why isn't Alex' text tokenizing")

We are getting the next output:

    ['why', 'isn', "'", 't', 'Alex', "'", 'text', 'token', '##izing']

I want to convert it to:

    why isn't Alex' text tokenizing


It seems the tokenizer doesn't do its job in the best way. If after tokenization we would have something like that:

    ['why', 'isn', "##'", '##t', 'Alex', "##'", 'text', 'token', '##izing']

It would be easy to convert:

    tokens = ['why', 'isn', "##'", '##t', 'Alex', "##'", 'text', 'token', '##izing']
    restored_text = [None] * len(tokens)
    tokens.extend(['#', '#'])
    for i in range(len(tokens )-2):
        if re.findall("#{2}", tokens[i+1]):
          restored_text [i] = tokens[i] + tokens[i+1].replace('##', '')
          if re.findall("#{2}", tokens [i+2]):
            restored_text[i] = restored_text[i] + tokens[i+2].replace('##', '')
        else:
          restored_text[i] = tokens[i]
    
    restored_text_without_masks = []
    for i in range(len(restored_text)):
        if not restored_text[i].startswith('#'):
            restored_text_without_masks.append(restored_text[i])

Comments:
I came across the same problem some days ago. I like your code, however it can be faster:

```python

def is_subtoken(word):
    if word[:2] == "##":
        return True
    else:
        return False

tokens = ['why', 'isn', "##'", '##t', 'Alex', "##'", 'text', 'token', '##izing']
restored_text = []
for i in range(len(tokens)):
    if not is_subtoken(tokens[i]) and (i+1)<len(tokens) and is_subtoken(tokens[i+1]):
        restored_text.append(tokens[i] + tokens[i+1][2:])
        if (i+2)<len(tokens) and is_subtoken(tokens[i+2]):
            restored_text[-1] = restored_text[-1] + tokens[i+2][2:]
    elif not is_subtoken(tokens[i]):
        restored_text.append(tokens[i])
```

@GuillemGSubies Did you solve this problem in your task?

> @GuillemGSubies Did you solve this problem in your task?

I used a modification of the code I posted above (in my use case I needed to interact with some NER labels also). However if you want to detokenize I think your code works perfectly.

@GuillemGSubies 
I just want to clarify something in my question.

Let's consider two sentences:

    "why isn't Alex's text tokenizing? The house on the left is the Smiths' house"

Now let's tokenize and decode:

    from transformers import BertTokenizer
    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
    tokenizer.decode(tokenizer.convert_tokens_to_ids(tokenizer.tokenize("why isn't Alex's text tokenizing? The house on the left is the Smiths' house")))

We get:

    "why isn't alex's text tokenizing? the house on the left is the smiths'house"



**My question is how dealing with missing space in some possessives like *smiths'house*?**


For me, it seems that the process of tokenization in Transformers is done not right. Let's consider output of

    tokenizer.tokenize("why isn't Alex's text tokenizing? The house on the left is the Smiths' house")

we get:

    ['why', 'isn', "'", 't', 'alex', "'", 's', 'text', 'token', '##izing', '?', 'the', 'house', 'on', 'the', 'left', 'is', 'the', 'smith', '##s', "'", 'house']

So in this step, we already have lost important information about the last apostrophe. It would be much better if tokenization was done in the another way:

    ['why', 'isn', "##'", '##t', 'alex', "##'", '##s', 'text', 'token', '##izing', '?', 'the', 'house', 'on', 'the', 'left', 'is', 'the', 'smith', '##s', "##'", 'house']

In this way, tokenization keeps all information about apostrophes, and we will not have problems with possessives.


This goes beyond of my understanding of the library, I am not a dev, sorry

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


still happens

cc @ArthurZucker for reference

@bauerem could you elaborate? The issue was closed 4 years ago. 
```
In [2]: from transformers import BertTokenizer

In [3]: tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
   ...: tokens = tokenizer.tokenize("why isn't Alex' text tokenizing")

In [4]: tokens
Out[4]: ['why', 'isn', "'", 't', 'alex', "'", 'text', 'token', '##izing']

In [5]: tokenizer.convert_tokens_to_string(tokens)
Out[5]: "why isn ' t alex ' text tokenizing"
```
seems alright to me

@ArthurZucker I think the most demonstrative example is the one in [this comment](https://github.com/huggingface/transformers/issues/3434#issuecomment-607947830) - specifically relating to the possessive apostrophe of `smiths'` - I'm able to replicate on main. 

Alright!
I am also unsure of why the tokenization here is "failing". It should be 1-1 equivalent to the one proposed in the bert paper. 

The decoding however might not be the best:

```python3
tokenizer.convert_tokens_to_string(tokenizer.tokenize("why isn't Alex's text tokenizing? The house on the left is the Smiths' house"))
```
does not lose the information on `'`, but the `decode` function treats `alex ' s` the same way it treats `smiths ' home`. 
```python3
In [37]: tokenizer.decode(tokenizer.encode("why isn't Alex's text tokenizing? The house on the left is the Smiths' house"), clean_up_tokenization_spaces=False)
Out[37]: "[CLS] why isn ' t alex ' s text tokenizing ? the house on the left is the smiths ' house [SEP]"
```
The issue is the `clean_up_tokenization_spaces`.:
```
    @staticmethod
    def clean_up_tokenization(out_string: str) -> str:
        """
        Clean up a list of simple English tokenization artifacts like spaces before punctuations and abbreviated forms.

        Args:
            out_string (`str`): The text to clean up.

        Returns:
            `str`: The cleaned-up string.
        """
        out_string = (
            out_string.replace(" .", ".")
            .replace(" ?", "?")
            .replace(" !", "!")
            .replace(" ,", ",")
            .replace(" ' ", "'")
            .replace(" n't", "n't")
            .replace(" 'm", "'m")
            .replace(" 's", "'s")
            .replace(" 've", "'ve")
            .replace(" 're", "'re")
        )
        return out_string
```
and specifically `            .replace(" ' ", "'")`. 

We could fix this using more complex regex, I would just not recommend using this as it's rule based and does not really preserved things. The bert tokenizer is already breaking the inpuut preventing input == output