gpt2-medium fine-tuned model.generate joins words and sentences together without space or newline
Author: albertbn
Pull request: no
Labels: name: wontfix

name: fp16

Hi, 

I have successfully fine-tuned and used a gpt2 model to generate text. My training corpus consist of short sentences - 3-5 words and longer ones 10-15 words. All separated by new line character. Sometimes ending with [ . ! ? ] sometimes not
  
`outputs = model.generate(
            input_ids=input_ids, max_length=max_length, temperature=temperature, 
            repetition_penalty=repetition_penalty,
            bos_token_id=tokenizer.bos_token_id,
            top_k=top_k,
            top_p=top_p
        )`

`ret = tokenizer.decode(outputs[0], skip_special_tokens=True)`

Then I fine-tuned a gpt2-medium model. The training corpus was slightly different, but structured the same as described above. 

I had to use --fp16 and --block_size=512 to fit in the GPU memory limits. 

The result:
using the fine-tuned a gpt2-medium model, I am experiencing a couple of issues:

1. I get frequent issues with lines or words stuck, without any new line or space: 
example:
**word1Word2Word3**
or:
**line 1 with some words!Another line with some words™️Next line...**

2. I get a 'warning': 
Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence

I've tried playing with the decode parameters with no luck:
`ret = tokenizer.decode(outputs[0], skip_special_tokens=False, clean_up_tokenization_spaces=False)`

Help appreciated, 

thanks in advance,
Albert


Comments:
I can provide a google colab notebook, the fine-tuned model, the training data (or whatever needed) to show the issue. I hope it's a tokenizer issue rather than a training fault on my side, since re-training would cost a lot of cash (the training data is quite big - ~22 million lines)

 

I'm suspecting `fp16` to be the reason. Not sure whether this is supported for `generation` yet. @sshleifer - do you know more about this maybe?

A colab notebook would be great though. Or even better would be if you could upload your model to the community models :-) 
This would make it very easy for us to find the bug:
https://huggingface.co/transformers/model_sharing.html

Yeah some sort of sharing to diagnose. I don't think fp16 is the problem. What does `outputs[0]` look like? 

hi, 

thanks for your reply, 

[https://huggingface.co/albertbn/gpt2-medium-finetuned-ads-fp16-blocksz512](url)

the above is the model, 

you can re-create the error using the following:

```
import torch
from transformers import GPT2Tokenizer, GPT2LMHeadModel

tokenizer = GPT2Tokenizer.from_pretrained(path)
model = GPT2LMHeadModel.from_pretrained(path)

if torch.cuda.is_available():
    model.to('cuda')

input_context = '''Find a plumber nearby!'''
input_ids = torch.tensor(tokenizer.encode(input_context)).unsqueeze(0)  
if torch.cuda.is_available():
    input_ids = input_ids.cuda()

max_length=150; temperature=.175; repetition_penalty=1.3; top_k=70; top_p=0.67
outputs = model.generate(
            input_ids=input_ids, max_length=max_length, temperature=temperature, repetition_penalty=repetition_penalty,
            bos_token_id=tokenizer.bos_token_id,
            top_k=top_k,
            top_p=top_p
        )

ret = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(ret)

# Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence
# Find a plumber nearby!
# Plumbing Services in Wellington NZ.
# 24/7 Emergency Plumbers Near You, Call Now For Fast Service or Repair of Your Plumbing!Need to Fix Leaking Pipes?‎NZ's #1 Gasfitter™️Call Us Today for Expert Advice & The Best Service!Get the Right Gasfitting Solution for your Home. Get It Installed Now - Free Quote Here !{KeyWord:Gas Fitting Installation}Quick And Efficient Installers

```

you can see the issue (lines stuck without \n) in the last line, starting with: 24/7 Emergency Plumbers...

thank you in advance,
Albert

> Yeah some sort of sharing to diagnose. I don't think fp16 is the problem. What does `outputs[0]` look like?

outputs[0] for the example I've posted looks like this:

```
tensor([16742,   257,   458,  4494,  6716,     0,   198,  3646, 28149,  6168,
          287, 30597, 26905,    13,   198,  1731,    14,    22, 18154,  1345,
        17024, 20173,   921,    11,  4889,  2735,  1114, 12549,  4809,   393,
        28912,   286,  3406,  1345, 28149,     0, 23037,   284, 13268,  1004,
          868,   350, 18636,    30, 48261, 37371,   338,  1303,    16, 14345,
           69,  1967,  8151, 37929, 14134,  4021,  6288,   329, 25516, 42708,
         1222,   383,  6705,  4809,     0,  3855,   262,  6498, 14345, 32232,
        28186,   329,   534,  5995,    13,  3497,   632,  2262,  4262,  2735,
          532,  3232, 19879,  3423,  5145,    90,  9218, 26449,    25, 39699,
          376,  2535, 32588,    92, 21063,   843,   412,  5632, 15545,   364],
       device='cuda:0')
```
there is no white space separating 0, 23037 (23037 is the only index in the output: Plumbing**!Need** )


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Regarding the second question

>     2. I get a 'warning':
>        Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence

It's explained [here](https://jaketae.github.io/study/gpt2/#setup): "For open-end generation, HuggingFace will set the padding token ID to be equal to the end-of-sentence token ID". Code is here: https://github.com/huggingface/transformers/blob/b880508440f43f80e35a78ccd2a32f3bde91cb23/src/transformers/generation_utils.py#L410-L414