PegasusForConditionalGeneration stops at unknown token
Author: adjeiv
Pull request: no
Labels: name: seq2seq

Hi all,

When following the code snippet from the [huggingface documentation](https://huggingface.co/transformers/master/model_doc/pegasus.html) but replacing the text, I have found that the summary stops when it reaches an unknown token. Is there a way around this?

src_text = [
"""As a child, the documentary maker, who was born in the Indian state of Kerala but now lives in Toronto, saw ceremonial elephants being paraded and thought they were beautiful. Later, she learned about the ordeal the animals are subjected to. "So many elephants had ghastly wounds on their hips, massive tumours and blood oozing out of their ankles, because chains had cut into their flesh and many of them were blind," Iyer told the BBC. She has made a documentary, Gods in Shackles, in an attempt to draw attention to the treatment of temple elephants she saw in India. "They were so helpless and the chains were so heavy," she said. "It was absolutely heart-breaking for me to witness this."""
]

Using this sample text, for example, my summary is:
As a child, the documentary maker, who was born in the Indian state of Kerala but now lives in Toronto, saw ceremonial elephants being paraded and thought they were beautiful, but later, she learned about the ordeal the animals are subjected to in India, in an attempt to draw attention to the treatment of templeunk_9

Comments:
The easiest way I can think of is to avoid generating the unk token altogether.

add the following method to `PegasusForConditionalGeneration`

```python
    def adjust_logits_during_generation(self, logits, cur_len, max_length):
        # Note, this will break with a tokenizer that is not PegasusTokenizer
        logits[:, list(range(2, 105))] = float("-inf")  # never predict unk tokens
        if cur_len == max_length - 1 and self.config.eos_token_id is not None:
            self._force_token_ids_generation(logits, self.config.eos_token_id)
        return logits
```

Let me know if that helps!



> The easiest way I can think of is to avoid generating the unk token altogether.
> 
> add the following method to `PegasusForConditionalGeneration`
> 
> ```python
>     def adjust_logits_during_generation(self, logits, cur_len, max_length):
>         # Note, this will break with a tokenizer that is not PegasusTokenizer
>         logits[:, list(range(2, 105))] = float("-inf")  # never predict unk tokens
>         if cur_len == max_length - 1 and self.config.eos_token_id is not None:
>             self._force_token_ids_generation(logits, self.config.eos_token_id)
>         return logits
> ```
> 
> Let me know if that helps!

Thank you for the reply!
I've added the code and checked it's being run, but unfortunately the output still stops at an unknown token.

I can't replicate.
got 
> As a child, the documentary maker, who was born in the Indian state of Kerala but now lives in Toronto, saw ceremonial elephants being paraded and thought they were beautiful.

on the branch of #7014 


Setting the min_length parameter to 100 yields the problem (as I should have mentioned). Might this be an issue with the minimum length being too long relative to the size of the input?

Yeah. the `xsum` model especially is trained to generate very short summaries.
`pegasus-arxiv` for example, can generate up to 256 tokens.

you can see each available checkpoint and it's maximum input and output sizes [here](https://github.com/huggingface/transformers/blob/0f58903bb62870342eae52f5a02c9105ec6f9b1e/src/transformers/configuration_pegasus.py#L50)

+ `max_length`: max length to generate
+ `max_position_embeddings`: max input size.