

TITLE
`prefix_allowed_tokens_fn` do not constrain when all allowed tokens have scores of `-inf`

COMMENTS
1

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
### System Info

transformer 4.25.1
python 3.8.16

### Who can help?

@gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When using `generate()` with `prefix_allowed_tokens_fn`, (more precisely, when using `PrefixConstrainedLogitsProcessor`),
when all tokens returned by `prefix_allowed_tokens_fn` have scores of `-inf`, the model does not comply with the constraints and picks the token which is not on the allowed token list.

### Expected behavior

Even if all allowed tokens have score of `-inf`, the model should pick tokens from allowed token list by `prefix_allowed_tokens_fn`.
I think it can be solved by using some clamp function or adding epsilon value to this code.
https://github.com/huggingface/transformers/blob/04ab5605fbb4ef207b10bf2772d88c53fc242e83/src/transformers/generation/logits_process.py#L692-L698

This is my own code to solve it.
However, it might cause other bugs.

```python
    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor) -> torch.FloatTensor:
        masked_score = torch.full_like(scores, -math.inf)
        for batch_id, beam_sent in enumerate(input_ids.view(-1, self._num_beams, input_ids.shape[-1])):
            for beam_id, sent in enumerate(beam_sent):
                allowed_idx = batch_id * self._num_beams + beam_id, self._prefix_allowed_tokens_fn(batch_id, sent)
                filtered_scores = torch.clamp(scores[allowed_idx], min=-10 ** 6)
                masked_score[allowed_idx] = filtered_scores
        return masked_score
```
Edit: The model works well on `torch.clamp()` with `min=-10 ** 6`, not `min=-10 ** 8`, when all allowed token's score is -inf. Too low score token in the sequence may have affected the decoding step. I updated the above code.