issue is, that ##string will repeats at intermediate, it collapses all index for mask words
Author: MuruganR96
Pull request: no

```
----------------------------------> how much belan i havin my credit card and also debitcard
----------------------------------> ['how', 'much', 'belan', 'i', 'havin', 'my', 'credit', 'card', 'and', 'also', 'debitcard']
----------------------------------> ['**belan**', '**havin**']
----------------------------------> [2, 4]
----------------------------------> ['how', 'much', '**belan**', 'i', '**havin**', 'my', 'credit', 'card', 'and', 'also', 'debitcard']
----------------------------------> how much belan i havin my credit card and also debitcard
before_tokenized_text-------------> ['how', 'much', **'bela'**, **'##n'**, 'i', **'ha'**, **'##vin'**, 'my', 'credit', 'card', 'and', 'also', '**de'**, **'##bit',** '**##card']**
index_useless---------------------> [2, 4]
after_tokenized_text--------------> ['how', 'much', '[MASK]', '##n', '[MASK]', 'ha', '##vin', 'my', 'credit', 'card', 'and', 'also', 'de', '##bit', '##card']
##########                    ['more', 'most']
##########                  2 <---------index_useless_length
##########                  2 <---------predicted_words_len
##########                  how much [MASK] n [MASK] ha vin my credit card and also de bit card <---------tokenized_text
##########   index_tk_aft                 [2, 4]
##########                    how much more n most ha vin my credit card and also de bit card
##########                   how much more n most ha vin my credit card and also de bit card <---------Result

```
i think As you understood. that spelling mistake words [2, 4] as Masking to predict. 
but in this place, what happened,

##string ->  '##n' ,  '##vin', like this spoil the predict final output.
i found and try so many ways. but all useless still.

**how to predict and fetch two more masking words?**

Thanks.


Comments:
Hi, I don't think you can do that in a clean way, sorry. That how BERT is trained.

> That how BERT is trained

i was pretrained our **bert-base-uncased** model with own dataset. 
Batch_size=32
max_seq_length=128

> I don't think you can do that in a clean way

you asked me "what was the problem now?" i think so.
normal word (ex: 'cadd')  splits into multiple ##string (ex: 'cad', '##d').
it was affected my mask word output.(like index dynamically changed)

step 1: tokenize 
tokenized_text = tokenizer.tokenize(source_transcript)

step 2:replace masked word as "[MASK]" in index position
tokenized_text[index] ='[MASK]'

step 3:predicting mask word
predicted_index = torch.argmax(predictions[0, masked_index]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]

ex:
original text:how to apply for **cadd** (['how', 'to', 'apply', 'for', 'cadd'])
masked word: cadd
index:4

step 1:
['how', 'le', 'comply', 'for', 'cad', '##d']
note: index length increased

step 2:
['how', 'to', 'apply', 'for', '[MASK]', '##d']
note: '[MASK]' index increased

step 3:
result: ['sb']
how to apply for **sb d**

actually i was pretrained model for this as "how to apply for card"
but it was not predicting well.

Main issue is, that ##string will repeats at intermediate, it collapses all index for mask words.

And this **Two to Three mask word prediction at the same sentence also very complex**.
 
`Two to Three mask word prediction at the same sentence also very complex
`

how to solve this problem @thomwolf  sir

> > That how BERT is trained
> 
> i was pretrained our **bert-base-uncased** model with own dataset.
> Batch_size=32
> max_seq_length=128
> 
> > I don't think you can do that in a clean way
> 
> you asked me "what was the problem now?" i think so.
> normal word (ex: 'cadd') splits into multiple ##string (ex: 'cad', '##d').
> it was affected my mask word output.(like index dynamically changed)
> 
> step 1: tokenize
> tokenized_text = tokenizer.tokenize(source_transcript)
> 
> step 2:replace masked word as "[MASK]" in index position
> tokenized_text[index] ='[MASK]'
> 
> step 3:predicting mask word
> predicted_index = torch.argmax(predictions[0, masked_index]).item()
> predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
> 
> ex:
> original text:how to apply for **cadd** (['how', 'to', 'apply', 'for', 'cadd'])
> masked word: cadd
> index:4
> 
> step 1:
> ['how', 'le', 'comply', 'for', 'cad', '##d']
> note: index length increased
> 
> step 2:
> ['how', 'to', 'apply', 'for', '[MASK]', '##d']
> note: '[MASK]' index increased
> 
> step 3:
> result: ['sb']
> how to apply for **sb d**
> 
> actually i was pretrained model for this as "how to apply for card"
> but it was not predicting well.
> 
> Main issue is, that ##string will repeats at intermediate, it collapses all index for mask words.
> 
> And this **Two to Three mask word prediction at the same sentence also very complex**.
> 
> `Two to Three mask word prediction at the same sentence also very complex `
> 
> how to solve this problem @thomwolf sir

sir @thomwolf  any suggestions.

Thanks.