GPT2 generating repetitive text
Author: DEBADRIBASAK
Pull request: no
Labels: name: wontfix

I was trying to use the pretrained GPT2LMHeadModel for generating texts by feeding some initial English words. But it is always generating repetitive texts.
Input: All
Output: All     All the same, the same, the same, the same, the same, the same, the same, the same, the same, the same, the same, the same,

Here is my code:
`import numpy as np
import pandas as pd
import torch
import torch.nn as nn
import torch.nn.functional as F
from pytorch_pretrained_BERT.pytorch_pretrained_bert.modeling_gpt2 import GPT2LMHeadModel
from pytorch_pretrained_BERT.pytorch_pretrained_bert.tokenization_gpt2 import GPT2Tokenizer
from pytorch_pretrained_BERT.pytorch_pretrained_bert.optimization_openai import OpenAIAdam
from tqdm import tqdm
import torch.optim as optim
import random
import time
import os
import sys
import random
import argparse
from pathlib import Path
from torch.utils.data import Dataset,TensorDataset,DataLoader,SequentialSampler,RandomSampler

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained("gpt2")
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
df = pd.read_csv("gpt_test.csv",sep="\t")
df = df.values

#context = tokenizer.encode(context)

model.to(device)
source = []
generated = []

for l in tqdm(range(len(df))):
    source.append(str(df[l,-2]))
    context = tokenizer.encode(str(df[l,-2]))
    past = None
    for i in range(40):
        input_ids = torch.tensor([context])
        input_ids = input_ids.to(device)
        pred,_ = model(input_ids=input_ids)
        predictions = torch.argmax(pred[0,-1,:]).item()
        context.append(predictions)
        if(predictions==2):
            break
    generated_text = tokenizer.decode(context)
    generated.append(generated_text)
    
df1 = pd.DataFrame({'Source': source,'Generated': generated})
df1.to_csv("./result_with_gpt.csv",sep="\t")`
Can someone point out the mistake? I will be highly grateful if the response is fast.

Comments:
Have you tried the provided GPT-2 generation example? It's here: https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_gpt2.py

> Have you tried the provided GPT-2 generation example? It's here: https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_gpt2.py

Hey, I encountered the same issue. I tried using the example you provided but it tends to produce repetitive text much more often than earlier versions of the library as well (from around 1-2 months back).  Thank you very much for all the work! 

> Have you tried the provided GPT-2 generation example? It's here: https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_gpt2.py

I tried the example. It is working properly. I think with `torch.argmax` there is chance of repetitive text generation. If we sample using `torch.multinomial`, there is always some variation.




`torch.argmax` is basically top_k with 1 which is very bad for creating "human like" sentences. A better way to sample is using Nucleus Sampling [https://arxiv.org/abs/1904.09751](url).
Not sure whether this is implemented in PyTorch yet.
EDIT: I have found this code from @thomwolf [https://gist.github.com/thomwolf/1a5a29f6962089e871b94cbd09daf317](url) that implements Nucleus sampling.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


 generating repetitive text when using GPU, but this not happens when using CPU, anyone knows how to solve this weird issue?

> when using GPU, but this not happens when using CPU, anyone knows how to solve this weird issue?

I am facing the same issue. Can someone lead in this problem?