T5 unsupervised training
Author: amlarraz
Pull request: no

I want to train T5 in a new language from scratch an I think the best way to do this is through the unsupervised denoising task (because you can have all text you want! no labels required! hurra!) 

However I have some doubts, I've seen [the HuggingFace  documentation about this](https://huggingface.co/transformers/model_doc/t5.html#training) and I wonder how to create the train data, I mean, is there any function in the library to add the sentinel tokens? In my own research I've worked with [the original T5 library](https://github.com/google-research/text-to-text-transfer-transformer) and I've seen that in this library they have some functions to do that, but this functions not apply the sentinel_tokens to the text but it replace the "noising" tokens by " always. 

My questions are:

1. It exists some functions to do that in HuggingFace?
2.  If the question 1 answer is not, anybody have any?
3. If the question 1 and 2 answers are not, What are the rules I must follow to create this function? Rules like:
           - How much concatenated tokens must the sentinel tokens mask? I mean, in the sentence "The cute dog walks in the park” they put the sentinel tokens in "cute dog" and my question is why this words election? I can take 1 token forever?
           - In each sentence I must start using the sentinel token number 1?

Thank you in advance! :)

Comments:
__UPDATE:__

I'm trying to make my own masking_function for this task. According with the T5 original paper, if you have two consecutive tokens to masking you must mask them using only one sentinel token so I need a function that searches the consecutive tokens ("rachas" in my language) in the random indices choosed. Here you have my code:

```
def racha_detection(lista):
    # It returns a list of lists where each sub-list contains the consecutive tokens in the list
    rachas = []
    racha = []
    for i, element in enumerate(lista):
        if (i<len(lista)-1) and (lista[i+1] == element+1):
            racha.append(element)
        else:
            if len(racha)>0:
                rachas.append(racha + [element])          
            else:# (i!=len(lista)-1):
                rachas.append([element])
            racha = []
    return rachas

def masking(tokenized_sentence, rachas):
    # Function to mask a tokenized_sentence (token ids) following the rachas described in rachas
    # Only one sentinel_token per racha
    sent_token_id = 0
    enmascared = tokenized_sentence.copy()
    for racha in rachas:
        sent_token = f'<extra_id_{sent_token_id}>'
        sent_id = tokenizer.encode(sent_token)[0]
        for i, idx in enumerate(racha):
            if i==0:
                enmascared[idx] = sent_id
            else:
                enmascared[idx] = -100
        sent_token_id += 1
        
    enmascared = [t for t in enmascared if t!=-100] 

    return enmascared

def add_noise(sentence, tokenizer, percent=0.15):
    # Function that takes a sentence, tokenizer and a noise percentage and returns
    # the masked input_ids and masked target_ids accordling with the T5 paper and HuggingFace docs
    # To see the process working uncomment all the prints ;)
    tokenized_sentence = tokenizer.encode(sentence)
    #print('PRE-MASKED:')
    #print('INPUT: {}'.format(tokenizer.convert_ids_to_tokens(tokenized_sentence)))
   
    idxs_2_mask = sorted(random.sample(range(len(tokenized_sentence)), 
                                       int(len(tokenized_sentence)*percent)))
    rachas = racha_detection(idxs_2_mask)
    enmascared_input = masking(tokenized_sentence, rachas)
    #print('RACHAS INPUT: {}'.format(rachas))
    idxs_2_mask = [idx for idx in range(len(tokenized_sentence)) if idx not in idxs_2_mask]
    rachas = racha_detection(idxs_2_mask)
    enmascared_target = masking(tokenized_sentence, rachas)
    #print('RACHAS TARGET: {}'.format(rachas))
    
    #print('POST-MASKED:')
    #print('INPUT: {}'.format(tokenizer.convert_ids_to_tokens(enmascared_input)))
    #print('TARGET: {}'.format(tokenizer.convert_ids_to_tokens(enmascared_target)))

    return enmascared_input, enmascared_target
```

I dont know if it is correct but it generates sequences like the sequences in the examples. What do you think?

Another question comes to my mind, is it neccesary to add the pad token at the beggining of the label in this task too? I'm using the "labels" argument to add the targets_ids to the model, I mean:

`model(input_ids=input_ids, labels=target_ids)`

Thank you in advance!

Hey @amlarraz,

As far as I know there is no pre-written function or script for unsupervised "sentinel masking" for T5. But it shouldn't be too difficult to do so. The innovation of T5's sentinel masking is exactly that you can mask multiple tokens with a single masking token which has been shown to yield better results as norrmal single token masking (a la BERT). 

So to answer your questions:
1) The data should be pre-processed as described in the paper and in the example in the docs, here: https://huggingface.co/transformers/model_doc/t5.html#training . The forum: http://discuss.huggingface.co/ is probably a better place to ask more specific questions about your code.

2) You don't need to add a padding token to the labels - this is done automatically here: https://github.com/huggingface/transformers/blob/2977bd528f06bada54afcf740219e65afd1c0883/src/transformers/modeling_t5.py#L638

Hi @patrickvonplaten !

Many thanks for answer me. As you said I've moved the question to the huggingface forum. If there are somebody interested in follow this topic this is the[ link to the conversation.](https://discuss.huggingface.co/t/train-t5-from-scratch/1781?u=amlarraz)

is there any update on your project?