How to structure input data for training TFGPT2LMHeadModel using model.fit() in TF2.0?
Author: brandonbell11
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I am able to use the run_lm_finetuning.py script easily, but I wish to be able to use TF2.0 and call model.fit() on distilgpt2. 

using the fine-tuning script as an example,  I structured my dataset as such:
```
#split text file by lines, tokenize lines, convert tokenized lines to integers
examples = []
with open(file_path, encoding="utf-8") as f:
    text = f.readlines()
for line in text:
    examples.append(tokenizer.encode(line))
    
#pad examples to appropriate length
pad_examples = tf.keras.preprocessing.sequence.pad_sequences(examples,
                                                             maxlen=256,padding='post',
                                                             truncating='post',
                                                             value=tokenizer.pad_token_id)

#create dataset, in the finetuning script, labels were just copies of the "examples" input array
dataset = tf.data.Dataset.from_tensor_slices((pad_examples, pad_examples))

BATCH_SIZE = 4
BUFFER_SIZE = 10000

dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)
```

However, when I compile and run the script like this:
```
optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=[loss, None, None, None, None, None, None], metrics=[metric])
model.fit(dataset, epochs=3)
```

my loss is NaN, and no training is happening.

I believe I am missing something fundamental about how GPT2 works with inputs/labels, but it seems from the torch script the labels and inputs are the same array and calling model.train() works just fine.

Any ideas would be greatly appreciated, as I have an existing TF2.0 architecture I am trying to connect to GPT-2 and being able to call model.fit() would be preferable. I have no issues fine-tuning BERT in TF2.0 with model.fit() as it is much clearer what the inputs and labels are in that case. 

Comments:
Hi, there is a fundamental difference between PyTorch and TensorFlow in that the losses for PyTorch can be computed both inside the model forward method as well as outside, whereas it is  only outside for TensorFlow.

This makes a difference when comparing the torch script and keras fit, as our GPT-2 implementation automatically computes the loss when giving the `labels` argument to the model, which would not be the case for TensorFlow. It computes the loss by shifting the examples by one index and comparing the model's token prediction to the true token value.

In order to train your GPT-2 (or DistilGPT-2) model on language modeling, you would have to create a dataset with:

- the examples
- the labels: these are the examples but shifted by one index so that the model may compare its prediction of the following token compared to the true token.

Let me know if you have additional questions.

@LysandreJik Thank you!

An additional question:
I have a text file with one sentence per line, and an end of sentence token at the end of each line. 

It seems I  should concatenate the text into one long string and sweep a "window" of a specified size over the text like so:
```
self.examples = []
            with open(file_path, encoding="utf-8") as f:
                text = f.read()

            tokenized_text = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(text))

            for i in range(0, len(tokenized_text)-block_size+1, block_size): # Truncate in block of block_size
                self.examples.append(tokenizer.build_inputs_with_special_tokens(tokenized_text[i:i+block_size]))
```
and so in order to get examples I would need to specify a length of tokens for training and my inputs would be `examples_chunk[:-1]` and my labels would be `examples_chunk[1:]` ?

Yes, you could also use `tokenizer.encode(text)`, rather than `tokenize` > `convert_tokens_to_ids` > `build_inputs_with_special_tokens`.

@LysandreJik 

Unfortunately I am still getting NaN loss and no training? Here is the code, which I assume is still not correct somehow but I cannot seem to figure out why. 

```
with open(file_path, encoding="utf-8") as f:
    text = f.read()
    
tokenized_text = tokenizer.encode(text)

examples = []
block_size = 100
for i in range(0, len(tokenized_text)-block_size+1, block_size): # Truncate in block of block_size
    examples.append(tokenized_text[i:i+block_size])

inputs, labels = [], []
for ex in examples:
    inputs.append(ex[:-1])
    labels.append(ex[1:])

dataset= tf.data.Dataset.from_tensor_slices((inputs,labels))

BATCH_SIZE = 16
BUFFER_SIZE = 10000

dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)

optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=[loss, None, None, None, None, None, None], metrics=[metric])
model.fit(dataset, epochs=3)
```

What is your input file? I'm running your code and I do get a decreasing loss alongside an increasing accuracy. Here's a [gist](https://gist.github.com/LysandreJik/c958925768eb6a9a72609ea99561d1cb) with the self-contained code (the text is in the file), let me know if running this still outputs a NaN loss.

Thanks so much for your help. That code works just as you described with your text file, and also works fine when I use my own text file. 

I discovered that the issue is coming from adding special tokens to the tokenizer. 

My text file is made up of one sentence per line such as:

```
<start> this is an example sentence from my text file <end>
<start> this is line two of my file <end>
```

When I don't change the bos_token and eos_token, I get decreasing loss and increasing accuracy. 
Adding the following code is what results in a NaN loss:
```
#change eos and bos tokens
special_tokens_dict = {'bos_token':"<start>", 'eos_token':"<end>"}
tokenizer.add_special_tokens(special_tokens_dict)
```

Any idea why this could be? Thank you again for the help. 

EDIT: included the code with the problem block 

```
with open(file_path, encoding="utf-8") as f:
    text = f.read()

#change eos and bos tokens
special_tokens_dict = {'bos_token':"<start>", 'eos_token':"<end>"}
tokenizer.add_special_tokens(special_tokens_dict)

tokenized_text = tokenizer.encode(text)

examples = []
block_size = 100
for i in range(0, len(tokenized_text)-block_size+1, block_size): # Truncate in block of block_size
    examples.append(tokenized_text[i:i+block_size])

inputs, labels = [], []
for ex in examples:
    inputs.append(ex[:-1])
    labels.append(ex[1:])

dataset= tf.data.Dataset.from_tensor_slices((inputs,labels))

BATCH_SIZE = 16
BUFFER_SIZE = 10000

dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)

optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=[loss, None, None, None, None, None, None], metrics=[metric])
model.fit(dataset, epochs=3)
```

In following the documentation, it appears that any time I try and run the command :
`model.resize_token_embeddings(len(tokenizer))` I get a `NotImplementedError`

If, however, I assign the bos and eos tokens when I first create the tokenizer:
`tokenizer = GPT2Tokenizer.from_pretrained("distilgpt2", bos_token='<start>', eos_token='<end>')`

Training results in decreasing loss and increasing accuracy. I realized this thread has drifted quite a bit, so I would be happy to close this and start another tokenizer thread. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.