Finetuning Pegasus for summarization task
Author: banunitte
Pull request: no

i have been trying to fine tune Pegasus for summarization task, it worked fine without getting any error.
but when i tried to generate the summary i was getting only empty list as a output.
i am not able to figure it out, is anything wrong with my fine tuning script?

```py
def label_smoothed_nll_loss(lprobs, target, epsilon, ignore_index=-100):
    """From fairseq"""
    if target.dim() == lprobs.dim() - 1:
        target = target.unsqueeze(-1)
    nll_loss = -lprobs.gather(dim=-1, index=target)
    smooth_loss = -lprobs.sum(dim=-1, keepdim=True)
    if ignore_index is not None:
        pad_mask = target.eq(ignore_index)
        nll_loss.masked_fill_(pad_mask, 0.0)
        smooth_loss.masked_fill_(pad_mask, 0.0)
    else:
        nll_loss = nll_loss.squeeze(-1)
        smooth_loss = smooth_loss.squeeze(-1)

    nll_loss = nll_loss.sum()  # mean()? Scared to break other math.
    smooth_loss = smooth_loss.sum()
    eps_i = epsilon / lprobs.size(-1)
    loss = (1.0 - epsilon) * nll_loss + eps_i * smooth_loss
    return loss, nll_loss

no_decay = ["bias", "LayerNorm.weight"]
optimizer_grouped_parameters = [
        {
            "params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
            "weight_decay": weight_decay,
        },
        {"params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], "weight_decay": 0.0},
    ]
optimizer = AdamW(optimizer_grouped_parameters, lr=learning_rate, eps=adam_epsilon)
scheduler = get_linear_schedule_with_warmup(
        optimizer, num_warmup_steps=warmup_steps, num_training_steps=t_total
    )

pad_token_id = tokenizer.pad_token_id
epochs = 5
for epoc in range(epochs):
  t0 = time.time()
  print("")
  print('======== Epoch {} ========'.format(epoc+1))
  model.train()
  total_train_loss = 0
  for i,batch in enumerate(train_dataset):
    title = []
    body = []
    for item in batch['title'].numpy():
      title.append(item.decode('utf-8'))
    for item in batch['body'].numpy():
      body.append(item.decode('utf-8')) 

    batch_tokens = tokenizer.prepare_seq2seq_batch(body,title,max_length=320,max_target_length=60,truncation=True,padding='max_length').to(device)
    decoder_input_ids = shift_tokens_right(batch_tokens['labels'], pad_token_id)
    outputs = model(batch_tokens['input_ids'], attention_mask=batch_tokens['attention_mask'], decoder_input_ids=decoder_input_ids, use_cache=False)
    lm_logits = outputs[0]
    lprobs = torch.nn.functional.log_softmax(lm_logits, dim=-1)
    loss, nll_loss = label_smoothed_nll_loss(
                lprobs, batch_tokens['labels'],0.1, ignore_index=pad_token_id
            )
    total_train_loss += loss.item()
  
    optimizer.zero_grad()
    loss.backward()
    #torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step()
    scheduler.step()
```


Comments:
notebook link :
https://colab.research.google.com/drive/1c7G1WXE6mgl2rwA-VR7q8DAqmbVqB62m?usp=sharing#scrollTo=VRzl54I-5isw 

![isuue](https://user-images.githubusercontent.com/6847024/94158117-a0d83300-fe9f-11ea-84b4-feec3b3b0b6d.png)


Facing the same issue. A reply on this will be highly appreciated.

This might help! Though implementation documentation is in tensorflow

https://github.com/google-research/pegasus#finetuning-on-downstream-datasets

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.