How to mask lm_labels and compute loss? --- Finetune gpt2: masking the lm_labels with '-1' and padding increase the perplexity a lot! 
Author: fabrahman
Pull request: no
Labels: name: wontfix

Hi,

I wanted to finetune gpt2 in a seq2seq format. For that I followed the same approach used for convAI2 explained [here](https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313).
I followed two steps and each step highly increase the ppl.
1) first I masked part of the lm_labels (with '-1') which I didn't wanted to calculate loss for. Doing that I realized the ppl increased but then I also try second step below
2) I padded my input to a maximum length 256 and masked the padded indices as well. Doing that increases the valid ppl (while training) starting from ```perplexity = tensor(6.8035e+10)``` rather than ```perplexity = tensor(17.67)``` when I don't mask lm_labels for padded indices and parts of input sequence that I don't want to compute loss for.
This ppl difference is huge and I am not sure if I have to taken care of sth else beside just masking lm_labels with '-1'?

Here is how I modify ```run_lm_finetuning.py``` script to pad my samples to a same length:
```
def pad(x, padding, padding_length=128):
    return x + [padding] * (padding_length - len(x))


class TextDataset(Dataset):
    def __init__(self, tokenizer, file_path='train', block_size=512):
        assert os.path.isfile(file_path)
        directory, filename = os.path.split(file_path)
        cached_features_file = os.path.join(directory, 'cached_lm_padded' + str(block_size) + '_' + filename)

        if os.path.exists(cached_features_file):
            logger.info("Loading features from cached file %s", cached_features_file)
            with open(cached_features_file, 'rb') as handle:
                self.examples = pickle.load(handle)
        else:
            logger.info("Creating features from dataset file at %s", directory)

            self.examples = []
            with open(file_path, encoding="utf-8") as f:
                text = f.readlines()

            text = [i.strip() for i in text]
            for sent in text:
                tokenized_text = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(sent))

                if len(tokenized_text) > block_size:  # Truncate in block of block_size
                    self.examples.append(tokenizer.build_inputs_with_special_tokens(tokenized_text[:block_size]))
                else:
                    self.examples.append(pad(tokenized_text, tokenizer.convert_tokens_to_ids('<pad>'), block_size))
            logger.info("Saving features into cached file %s", cached_features_file)
            with open(cached_features_file, 'wb') as handle:
                pickle.dump(self.examples, handle, protocol=pickle.HIGHEST_PROTOCOL)

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, item):
        return torch.tensor(self.examples[item])
```

And this is how I mask the lm_labels, given labels =input_ids and start and end being my desired indices to mask all indices before start and after end (exclusive):

```
def mask_intervals(labels, start, end):
    x = labels.clone()

    for i in range(x.shape[0]):
        start_index = (labels[i, :] == start).nonzero()
        end_index = (labels[i, :] == end).nonzero()

        x[i, :start_index + 1] = -1
        x[i, end_index:] = -1
        
    return x
```
        
I want to know if masking with '-1' suffices or should I change some other part of the code regarding loss computation?

Comments:
Can any one help regarding this question? I really appreciate that. Is there any example of finetuning and masking the context and padded indices? I am following [this](https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313), but I get a uge perplexity compared to when I don't mask.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.