text: 
  
  TITLE
  T5 generations for pretraining objective degenerate
  
  COMMENTS
  3
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ## The issue
  
  I am using a pretrained T5 model to generate missing spans as in the pretraining objective. However, I'm finding that these generations deteriorate for longer sequences (usually after around the 25th span or so). Below is an example of this deterioration on a sequence (from the IMDB dataset) where 15% of the tokens have been randomly masked with sentinel tokens. Given that the T5 model was pretrained using sequences of up to 512 tokens with 15% of tokens masked, shouldn't it be possible to obtain good generations on sequences like the one below? Why are generations like this one deteriorating? Thank you!
  
  ## Environment info
  - `transformers` version: 3.5.0
  - Platform: Linux-4.15.0-45-generic-x86_64-with-debian-buster-sid
  - Python version: 3.7.7
  - PyTorch version (GPU?): 1.7.0 (True)
  - Tensorflow version (GPU?): not installed (NA)
  - Using GPU in script?: True
  
  ### Who can help
  
  @patrickvonplaten 
  
  ## To reproduce
  
  Steps to reproduce the behavior:
  
  ```python
  import torch
  from transformers import T5Tokenizer, T5ForConditionalGeneration
  
  device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
  
  model_name = "t5-base"
  t5_tokenizer = T5Tokenizer.from_pretrained(model_name)
  t5_model = T5ForConditionalGeneration.from_pretrained(model_name)
  t5_model = t5_model.to(device)
  
  original_sentence = "ROCK STAR is a well-told Hollywood-style rendition of the tale based on fact actually on how Ripper became Rob Halford's replacement for Judas Priest. Mark Wahlberg poured on his likable boy-ish charm and performed with believable admirably, something he has been known to do since the release of BOOGIE NIGHTS.  Stephen Herek, no stranger to musically-themed movies, takes the audience through the wonders of the breakneck lifestyle of an extinct species, the Hair-Metal Rock God. Wahlberg's 'Izzy' acts as the film's host plays the everyman who gets to see his wish come true. His likable character quickly wins over the heart of the viewer, who wants to see him succeed and gets the chance to give him the Metal 'goat horn' hand-sign several times over.  The only real complaint with the story is that the supporting cast, namely the other members of the band, were not fleshed out, or even introduced, properly. More interaction with these life-long Rock musicians would have amplified and solidified Izzy's new surroundings.   Naturally, ROCK STAR is filled with great music. Rabin's score, the Steel Dragon's original work and plenty of 80's-style Metal hits makes this soundtrack a must-have! Let's all hope that films like ROCK STAR not only give a credibility to a style of music that helped define a generation but also spark a very-needed revival.</s>"
  sentence = "ROCK STAR is a well-told Hollywood-style rendition<extra_id_0> the tale <extra_id_1> on<extra_id_2> on<extra_id_3> Ri<extra_id_4> became Rob Hal<extra_id_5>'s replacement for Ju<extra_id_6> Priest.<extra_id_7> Wahlberg poured on his likable boy-ish charm and performed with believable admirably, something<extra_id_8>he has been known to do<extra_id_9> the release of BOOGIE NIGHTS. Stephen Herek<extra_id_10> no stranger to musically-themed<extra_id_11>, takes the<extra_id_12> through the<extra_id_13>s<extra_id_14> break<extra_id_15> of<extra_id_16> extinct<extra_id_17>, the<extra_id_18>-Metal Rock<extra_id_19> Wahlberg's 'Izzy' acts as the film's host plays the everyman who gets to see his wish come true. His<extra_id_20>likable character quickly<extra_id_21> the heart of the viewer, who<extra_id_22> to see him succeed and gets the chance to give him the Metal 'goat horn' hand-sign several times over<extra_id_23> The only real complaint with the<extra_id_24> is that the supporting<extra_id_25>,<extra_id_26>namely the other members of<extra_id_27> band, were not fleshed out, or even introduced, properly<extra_id_28> More interaction with these life-long<extra_id_29> musicians would have amplified and solidified Izzy's new surroundings<extra_id_30> Naturally,<extra_id_31>CK STAR is filled<extra_id_32> great music. Rabin's score, the Steel Dragon<extra_id_33>s original work<extra_id_34> of 80's-style Metal<extra_id_35> makes this soundtrack<extra_id_36>a must-have<extra_id_37> all hope that films like ROCK STAR not only give a credibility<extra_id_38> a style of music that helped define a generation but also spark a very-needed revival<extra_id_39></s>"
  encoded = t5_tokenizer.encode(sentence)
  print("original sentence: ", original_sentence)
  print("\nmasked sentence: ", sentence)
  print("\nnum tokens masked sentence: ", len(encoded))
  encoded_tensor = torch.LongTensor(encoded).unsqueeze(0).to(device)
  eos_token_id = t5_tokenizer.encode("<extra_id_40>")[0]
  batch = t5_model.generate(encoded_tensor, early_stopping = True, max_length = 300, eos_token_id = eos_token_id, no_repeat_ngram_size = 2, num_beams = 1, num_return_sequences = 1)
  
  for b in batch:
      print("\noutput: ")
      print(t5_tokenizer.decode(b, skip_special_tokens = False))
  ```
  
  output:
  
  > original sentence:  ROCK STAR is a well-told Hollywood-style rendition of the tale based on fact actually on how Ripper became Rob Halford's replacement for Judas Priest. Mark Wahlberg poured on his likable boy-ish charm and performed with believable admirably, something he has been known to do since the release of BOOGIE NIGHTS.  Stephen Herek, no stranger to musically-themed movies, takes the audience through the wonders of the breakneck lifestyle of an extinct species, the Hair-Metal Rock God. Wahlberg's 'Izzy' acts as the film's host plays the everyman who gets to see his wish come true. His likable character quickly wins over the heart of the viewer, who wants to see him succeed and gets the chance to give him the Metal 'goat horn' hand-sign several times over.  The only real complaint with the story is that the supporting cast, namely the other members of the band, were not fleshed out, or even introduced, properly. More interaction with these life-long Rock musicians would have amplified and solidified Izzy's new surroundings.   Naturally, ROCK STAR is filled with great music. Rabin's score, the Steel Dragon's original work and plenty of 80's-style Metal hits makes this soundtrack a must-have! Let's all hope that films like ROCK STAR not only give a credibility to a style of music that helped define a generation but also spark a very-needed revival.</s>
  > 
  > masked sentence:  ROCK STAR is a well-told Hollywood-style rendition<extra_id_0> the tale <extra_id_1> on<extra_id_2> on<extra_id_3> Ri<extra_id_4> became Rob Hal<extra_id_5>'s replacement for Ju<extra_id_6> Priest.<extra_id_7> Wahlberg poured on his likable boy-ish charm and performed with believable admirably, something<extra_id_8>he has been known to do<extra_id_9> the release of BOOGIE NIGHTS. Stephen Herek<extra_id_10> no stranger to musically-themed<extra_id_11>, takes the<extra_id_12> through the<extra_id_13>s<extra_id_14> break<extra_id_15> of<extra_id_16> extinct<extra_id_17>, the<extra_id_18>-Metal Rock<extra_id_19> Wahlberg's 'Izzy' acts as the film's host plays the everyman who gets to see his wish come true. His<extra_id_20>likable character quickly<extra_id_21> the heart of the viewer, who<extra_id_22> to see him succeed and gets the chance to give him the Metal 'goat horn' hand-sign several times over<extra_id_23> The only real complaint with the<extra_id_24> is that the supporting<extra_id_25>,<extra_id_26>namely the other members of<extra_id_27> band, were not fleshed out, or even introduced, properly<extra_id_28> More interaction with these life-long<extra_id_29> musicians would have amplified and solidified Izzy's new surroundings<extra_id_30> Naturally,<extra_id_31>CK STAR is filled<extra_id_32> great music. Rabin's score, the Steel Dragon<extra_id_33>s original work<extra_id_34> of 80's-style Metal<extra_id_35> makes this soundtrack<extra_id_36>a must-have<extra_id_37> all hope that films like ROCK STAR not only give a credibility<extra_id_38> a style of music that helped define a generation but also spark a very-needed revival<extra_id_39></s>
  > 
  > num tokens masked sentence:  337
  > 
  > output: 
  > <extra_id_0> of<extra_id_1> of the man who<extra_id_2> the day of his death<extra_id_3> the<extra_id_4>m<extra_id_5>e<extra_id_6>das<extra_id_7> Steven<extra_id_8> that<extra_id_9> since<extra_id_10>o,<extra_id_11> films<extra_id_12> audience<extra_id_13> 'rock<extra_id_14>aga' of a<extra_id_15>-up<extra_id_16> the<extra_id_17> CK STAR band<extra_id_18> newest<extra_id_19> band. Steven<extra_id_20> incredibly<extra_id_21> captures<extra_id_22> is eager<extra_id_23>.<extra_id_24> film<extra_id_25> cast<extra_id_26> and<extra_id_27> the -Metal Rock, the band's sailor, and the members of Izzy' emcees, who were <extra_id_1><extra_id_20><extra_id_19>.-n " de an (s, in<extra_id_10> thetgrae and pro also le of'lr to ex si not on<extra_id_5><extra_id_3> I<extra_id_7>" ensemble­ ⁇ » be last for fiia =/ den?<extra_id_26> pour as --) 2 $: + 1 S un former dis spa<extra_id_17> tub root will at both second<extra_id_25> is no bout muscle hard des<extra_id_21>re<extra_id_23> baseball facialw mi& * [...;
  