Pegasus: IndexError: index out of range in self
Author: yxyzzz
Pull request: no

@sshleifer

## Environment info
- `transformers` version: 3.0.2
- Platform: macOS-10.14.6-x86_64-i386-64bit
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

## Information
Model I am using: google/pegasus-cnn_dailymail

The problem arises when using:
```
import torch
from transformers import AutoModelWithLMHead, AutoTokenizer

torch_device = 'cuda' if torch.cuda.is_available() else 'cpu'

model = AutoModelWithLMHead.from_pretrained("google/pegasus-cnn_dailymail")
tokenizer = AutoTokenizer.from_pretrained("google/pegasus-cnn_dailymail")

src_text = [
    '(CNN)James Best, best known for his portrayal of bumbling sheriff Rosco P. Coltrane on TV\'s "The Dukes of Hazzard," died Monday after a brief illness. He was 88. Best died in hospice in Hickory, North Carolina, of complications from pneumonia, said Steve Latshaw, a longtime friend and Hollywood colleague. Although he\'d been a busy actor for decades in theater and in Hollywood, Best didn\'t become famous until 1979, when "The Dukes of Hazzard\'s" cornpone charms began beaming into millions of American homes almost every Friday night. For seven seasons, Best\'s Rosco P. Coltrane chased the moonshine-running Duke boys back and forth across the back roads of fictitious Hazzard County, Georgia, although his "hot pursuit" usually ended with him crashing his patrol car. Although Rosco was slow-witted and corrupt, Best gave him a childlike enthusiasm that got laughs and made him endearing. His character became known for his distinctive "kew-kew-kew" chuckle and for goofy catchphrases such as "cuff \'em and stuff \'em!" upon making an arrest. Among the most popular shows on TV in the early \'80s, "The Dukes of Hazzard" ran until 1985 and spawned TV movies, an animated series and video games. Several of Best\'s "Hazzard" co-stars paid tribute to the late actor on social media. "I laughed and learned more from Jimmie in one hour than from anyone else in a whole year," co-star John Schneider, who played Bo Duke, said on Twitter. "Give Uncle Jesse my love when you see him dear friend." "Jimmy Best was the most constantly creative person I have ever known," said Ben Jones, who played mechanic Cooter on the show, in a Facebook post. "Every minute of his long life was spent acting, writing, producing, painting, teaching, fishing, or involved in another of his life\'s many passions." Born Jewel Guy on July 26, 1926, in Powderly, Kentucky, Best was orphaned at 3 and adopted by Armen and Essa Best, who renamed him James and raised him in rural Indiana. Best served in the Army during World War II before launching his acting career. In the 1950s and 1960s, he accumulated scores of credits, playing a range of colorful supporting characters in such TV shows as "The Twilight Zone," "Bonanza," "The Andy Griffith Show" and "Gunsmoke." He later appeared in a handful of Burt Reynolds\' movies, including "Hooper" and "The End." But Best will always be best known for his "Hazzard" role, which lives on in reruns. "Jimmie was my teacher, mentor, close friend and collaborator for 26 years," Latshaw said. "I directed two of his feature films, including the recent \'Return of the Killer Shrews,\' a sequel he co-wrote and was quite proud of as he had made the first one more than 50 years earlier." People we\'ve lost in 2015 . CNN\'s Stella Chan contributed to this story.'
]
batch = tokenizer.prepare_seq2seq_batch(src_text, truncation=True, padding='longest').to(torch_device)
translated = model.generate(**batch)
tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True)
```

The tasks I am working on is:
* abstractive text summarization

## To reproduce
1. Run the script above. The traceback:
```
IndexError                                Traceback (most recent call last)
<ipython-input-1-ac154ffeb574> in <module>
     13 ]
     14 batch = tokenizer.prepare_seq2seq_batch(src_text, truncation=True, padding='longest').to(torch_device)
---> 15 translated = model.generate(**batch)
     16 tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True)

~/anaconda3/envs/abstractive_summarizer/lib/python3.8/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     13         def decorate_context(*args, **kwargs):
     14             with self:
---> 15                 return func(*args, **kwargs)
     16         return decorate_context
     17 

~/projects/transformers/src/transformers/generation_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, num_return_sequences, attention_mask, decoder_start_token_id, use_cache, **model_specific_kwargs)
    394             encoder = self.get_encoder()
    395 
--> 396             encoder_outputs: tuple = encoder(input_ids, attention_mask=attention_mask)
    397 
    398         # Expand input ids if num_beams > 1 or num_return_sequences > 1

~/anaconda3/envs/abstractive_summarizer/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

~/projects/transformers/src/transformers/modeling_bart.py in forward(self, input_ids, attention_mask, output_attentions, output_hidden_states, return_dict)
    328 
    329         inputs_embeds = self.embed_tokens(input_ids) * self.embed_scale
--> 330         embed_pos = self.embed_positions(input_ids)
    331         x = inputs_embeds + embed_pos
    332         x = self.layernorm_embedding(x)

~/anaconda3/envs/abstractive_summarizer/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

~/anaconda3/envs/abstractive_summarizer/lib/python3.8/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     13         def decorate_context(*args, **kwargs):
     14             with self:
---> 15                 return func(*args, **kwargs)
     16         return decorate_context
     17 

~/projects/transformers/src/transformers/modeling_bart.py in forward(self, input_ids, use_cache)
   1337             # starts at 0, ends at 1-seq_len
   1338             positions = torch.arange(seq_len, dtype=torch.long, device=self.weight.device)
-> 1339         return super().forward(positions)

~/anaconda3/envs/abstractive_summarizer/lib/python3.8/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    122 
    123     def forward(self, input: Tensor) -> Tensor:
--> 124         return F.embedding(
    125             input, self.weight, self.padding_idx, self.max_norm,
    126             self.norm_type, self.scale_grad_by_freq, self.sparse)

~/anaconda3/envs/abstractive_summarizer/lib/python3.8/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1812         # remove once script supports set_grad_enabled
   1813         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1814     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1815 
   1816 

IndexError: index out of range in self
```

## Expected behavior
IndexError: index out of range in self


Comments:
Found bug, will be fix by EOD.

This is a coincidence. I found the same issue with pegasus-arxiv and pegasus-pubmed today. Glad this will be fixed

> 
> ## Information
> Model I am using: google/pegasus-cnn_dailymail
> 
> The problem arises when using:
> 
> ```
> import torch
> from transformers import AutoModelWithLMHead, AutoTokenizer
> 
> torch_device = 'cuda' if torch.cuda.is_available() else 'cpu'
> 
> model = AutoModelWithLMHead.from_pretrained("google/pegasus-cnn_dailymail")
> tokenizer = AutoTokenizer.from_pretrained("google/pegasus-cnn_dailymail")
> 
> src_text = [
>     '(CNN)James Best, best known for his portrayal of bumbling sheriff Rosco P. Coltrane on TV\'s "The Dukes of Hazzard," died Monday after a brief illness. He was 88. Best died in hospice in Hickory, North Carolina, of complications from pneumonia, said Steve Latshaw, a longtime friend and Hollywood colleague. Although he\'d been a busy actor for decades in theater and in Hollywood, Best didn\'t become famous until 1979, when "The Dukes of Hazzard\'s" cornpone charms began beaming into millions of American homes almost every Friday night. For seven seasons, Best\'s Rosco P. Coltrane chased the moonshine-running Duke boys back and forth across the back roads of fictitious Hazzard County, Georgia, although his "hot pursuit" usually ended with him crashing his patrol car. Although Rosco was slow-witted and corrupt, Best gave him a childlike enthusiasm that got laughs and made him endearing. His character became known for his distinctive "kew-kew-kew" chuckle and for goofy catchphrases such as "cuff \'em and stuff \'em!" upon making an arrest. Among the most popular shows on TV in the early \'80s, "The Dukes of Hazzard" ran until 1985 and spawned TV movies, an animated series and video games. Several of Best\'s "Hazzard" co-stars paid tribute to the late actor on social media. "I laughed and learned more from Jimmie in one hour than from anyone else in a whole year," co-star John Schneider, who played Bo Duke, said on Twitter. "Give Uncle Jesse my love when you see him dear friend." "Jimmy Best was the most constantly creative person I have ever known," said Ben Jones, who played mechanic Cooter on the show, in a Facebook post. "Every minute of his long life was spent acting, writing, producing, painting, teaching, fishing, or involved in another of his life\'s many passions." Born Jewel Guy on July 26, 1926, in Powderly, Kentucky, Best was orphaned at 3 and adopted by Armen and Essa Best, who renamed him James and raised him in rural Indiana. Best served in the Army during World War II before launching his acting career. In the 1950s and 1960s, he accumulated scores of credits, playing a range of colorful supporting characters in such TV shows as "The Twilight Zone," "Bonanza," "The Andy Griffith Show" and "Gunsmoke." He later appeared in a handful of Burt Reynolds\' movies, including "Hooper" and "The End." But Best will always be best known for his "Hazzard" role, which lives on in reruns. "Jimmie was my teacher, mentor, close friend and collaborator for 26 years," Latshaw said. "I directed two of his feature films, including the recent \'Return of the Killer Shrews,\' a sequel he co-wrote and was quite proud of as he had made the first one more than 50 years earlier." People we\'ve lost in 2015 . CNN\'s Stella Chan contributed to this story.'
> ]
> batch = tokenizer.prepare_seq2seq_batch(src_text, truncation=True, padding='longest').to(torch_device)
> translated = model.generate(**batch)
> tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True)
> ```
> 
Currently I am overcoming this by passing additional parameter max_length=512 in prepare_seq2seq_batch. 

This is not what I want. I want to expand the length to atleast 2048 if possible. But I don't see a way to do it

The bug should be fixed. Please try again.

The following max_lengths are recommended:
```
max_input_length = {
    "xsum": 512,
    "cnn_dailymail": 1024,
    "newsroom": 512,
    "wikihow": 512,
    "multi_news": 1024,
    "reddit_tifu": 512,
    "big_patent": 1024,
    "arxiv": 1024,
    "pubmed": 1024,
    "gigaword": 128,
    "aeslc": 512,
    "billsum": 1024,
    "large": 1024,
}
```

@suchig You can probably do longer with the current code by passing `max_position_embeddings=2048`, but no model was finetuned on sequences of that length so it might (a) OOM or (b) perform poorly.

> The bug should be fixed. Please try again.
> 
> The following max_lengths are recommended:
> 
> ```
> max_input_length = {
>     "xsum": 512,
>     "cnn_dailymail": 1024,
>     "newsroom": 512,
>     "wikihow": 512,
>     "multi_news": 1024,
>     "reddit_tifu": 512,
>     "big_patent": 1024,
>     "arxiv": 1024,
>     "pubmed": 1024,
>     "gigaword": 128,
>     "aeslc": 512,
>     "billsum": 1024,
>     "large": 1024,
> }
> ```
> 
> @suchig You can probably do longer with the current code by passing `max_position_embeddings=2048`, but no model was finetuned on sequences of that length so it might (a) OOM or (b) perform poorly.

At this junction, I am not even able to go beyond PegasusForConditionalGeneration.from_pretrained(mname) statement. It was working fine earlier in the morning. It feels as though something has changed in the pretrained model checkpoint this afternoon.

@suchig please stick to one issue. 

Hi @sshleifer , I want to a train a model for 2048 input length. My idea is to use a distilled teacher model and change the source length to 2048. How can I change the `max_position_embeddings`? 

I'm using the following command to train the model on CNN-DM dataset.

```python examples/seq2seq/distillation.py --teacher sshleifer/distill-pegasus-cnn-16-4 --data_dir ./data/cnn_dm/ --student_decoder_layers 3 --student_encoder_layers 12 --freeze_embeds --learning_rate=3e-4 --do_train --do_predict --val_check_interval 0.1 --n_val 1000 --eval_beams 2 --length_penalty=0.5 --val_metric rouge2 --max_source_length 2048 --max_target_length=142 --val_max_target_length=142 --test_max_target_length=142 --model_name_or_path IGNORED --alpha_hid=3. --train_batch_size=1 --eval_batch_size=1 --gradient_accumulation_steps=256 --sortish_sampler --num_train_epochs=6 --warmup_steps 500 --output_dir distilpegasus_xsum_12_3 --gpus 1 --tokenizer_name sshleifer/distill-pegasus-cnn-16-4 --num_workers=0 --adafactor```

That command will be very slow, but you can achieve your goal by passing `max_position_embeddings=2048` [here](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/distillation.py#L41)