Finetune and generate text with BertForMaskedLM
Author: vdperera
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I am trying to fine-tune and generate text using BertForMaskedLM. Although my script works I am not getting the output I am expecting. I am confused on what should I pass to BertForMaskedLM when training (attention mask, token types ids, etc) and how to generate text once the model is fine tuned. Any help is welcome, hereafter is my current code:

```
import torch
from torch.optim import Adam
from transformers import BertForMaskedLM, AutoTokenizer


if __name__ == "__main__":

    lr = 0.002
    epochs = 20
    model = BertForMaskedLM.from_pretrained("bert-base-uncased")
    tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
    optimizer = Adam(model.parameters(), lr=lr)

    dataset = ["this is the first sentence.",
               "this is the second, slightly longer, sentence.",
               "this is the third and last sentence."]

    # We precomputed this
    max_len = 12

    # Since we only have 3 sentences we fit all our dataset in a single batch.
    # The padded batch will look like:
    # [[101, 2023, 2003, 1996, 2034, 6251, 1012, 102, 0, 0, 0, 0],
    #  [101, 2023, 2003, 1996, 2117, 1010, 3621, 2936, 1010, 6251, 1012, 102],
    #  [101, 2023, 2003, 1996, 2353, 1998, 2197, 6251, 1012, 102, 0, 0]]
    padded_batch = []
    padding_id = tokenizer.convert_tokens_to_ids(tokenizer.pad_token)
    for sentence in dataset:
        encoded_sentence = tokenizer.encode(sentence)
        padded_sentence = encoded_sentence + \
            [padding_id]*(max_len-len(encoded_sentence))

        padded_batch.append(padded_sentence)

    # The attention mask will have the same shape of the batch, with 0s for the
    # padded element and 1s for the non-padded ones
    # [[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0],
    #  [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
    #  [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]]
    attention_mask = [[1 if t_id != padding_id else 0 for t_id in
                       tensor] for tensor in padded_batch]

    # The lm_labels will be the same as the padded batch but for the padded
    # elements which are replaced by -1
    # [[101, 2023, 2003, 1996, 2034, 6251, 1012, 102, -1, -1, -1, -1],
    #  [101, 2023, 2003, 1996, 2117, 1010, 3621, 2936, 1010, 6251, 1012, 102],
    #  [101, 2023, 2003, 1996, 2353, 1998, 2197, 6251, 1012, 102, -1, -1]]
    lm_labels = [[t_id if t_id != padding_id else -1 for t_id in tensor] for
                 tensor in padded_batch]

    # Converting the model input from list to tensor
    padded_batch = torch.tensor(padded_batch)
    attention_mask = torch.tensor(attention_mask)
    lm_labels = torch.tensor(lm_labels)

    # Since we only have one batch every epoch we do a single forward pass,
    # backprop and optimization step
    for i in range(epochs):
        loss, _ = model(input_ids=padded_batch, attention_mask=attention_mask,
                        lm_labels=lm_labels)
        print(loss.item())

        loss.backward()
        optimizer.step()
        model.zero_grad()

    # The model should now be trained and we want to generate the first three
    # words of a new sentence. Given the training data used we expect it to be
    # "this is the".

    # Initialize the model input with "[CLS] [MASK]" to generate word w_1, the
    # input for w_i is "[CLS] w_1 ... w_(i-1) [MASK]" where w_1 ... w_(i-1)
    # have been generated during previous steps.
    output = [101, 103]
    for i in range(4):
        generation_input = torch.tensor([output])
        pred = model(generation_input)[0]
        new_index = torch.argmax(pred, -1)
        output[-1] = new_index[:, -1].item()
        output.append(103)

    print(output)
    print(tokenizer.decode(output))

```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


did you find a solution?

Hi, I am also encountering this problem. Is it possible to please provide an example for fine tuning BertForMaskedLM on a specific type of text (ex Medical corpus) ? Especially what input should be passed to BertForMaskedLM to fine tune it (attention mask, token types ids, masked_token_index)?