[WIP] Fix edge cases in TopPLogitsWarper when top_p equals 0 or 1
Author: NinedayWang
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->
This PR fixes edge cases in TopPLogitsWarper when top_p equals 0 or 1. The following are specific contributions:
1. Make the code consistent with the `ValueError` error message.
2. Fix the `nan` problem. In the original PyTorch implementation, if `do_sample=True` and `top_p=0`, then each position in logits will be set to `-float("Inf")`, and then the `nan` problem will be encountered after softmax.
3. Make the three frameworks (PyTorch/TF/FLAX) have the same behavior. In the original implementation, when `do_sample=True` and `top_p=0`, the PyTorch framework would encounter `nan` errors, while TF and FLAX would keep the highest scoring token due to their right shift operations.



## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@gante @patrickvonplaten 




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19839). All of your documentation changes will be reflected on that endpoint.

Hi @NinedayWang 👋 

The edge cases are intentionally breaking -- `top_p=0.0` means in theory that no token can be sampled (and in practice, that they have the same odds), while `top_p=1.0` is equivalent to not having the `top_p` operation.

I struggle to see a use case for these changes, but I'm open to suggestions :)



> Hi @NinedayWang 👋
> 
> The edge cases are intentionally breaking -- `top_p=0.0` means in theory that no token can be sampled (and in practice, that they have the same odds), while `top_p=1.0` is equivalent to not having the `top_p` operation.
> 
> I struggle to see a use case for these changes, but I'm open to suggestions :)

Thanks for your quick reply. @gante 

I think if the changes are not made, there will be the following problems:
1. When `do_sample=True` and `top_p=0`:
(1) In PyTorch, `TopPLogitsWarper` will filter any logit to be `-inf`, causing us to get logits filled with `nan` after softmax, then the program will throw a `nan` exception on `multinomial` operation as follows:
	```
	Traceback (most recent call last):
	  File "eval_human_eval_wx.py", line 118, in <module>
	    evaluate_on_human_eval(
	  File "eval_human_eval_wx.py", line 98, in evaluate_on_human_eval
	    gen_results = run_code_generation(pipe, input_prompt, num_completions=generate_batch_size, **gen_kwargs)
	  File "eval_human_eval_wx.py", line 44, in run_code_generation
	    code_gens = pipe(prompt,
	  File "/data/miniconda3/envs/env-3.8.8/lib/python3.8/site-packages/transformers/pipelines/text_generation.py", line 187, in __call__
	    return super().__call__(text_inputs, **kwargs)
	  File "/data/miniconda3/envs/env-3.8.8/lib/python3.8/site-packages/transformers/pipelines/base.py", line 1074, in __call__
	    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
	  File "/data/miniconda3/envs/env-3.8.8/lib/python3.8/site-packages/transformers/pipelines/base.py", line 1081, in run_single
	    model_outputs = self.forward(model_inputs, **forward_params)
	  File "/data/miniconda3/envs/env-3.8.8/lib/python3.8/site-packages/transformers/pipelines/base.py", line 990, in forward
	    model_outputs = self._forward(model_inputs, **forward_params)
	  File "/data/miniconda3/envs/env-3.8.8/lib/python3.8/site-packages/transformers/pipelines/text_generation.py", line 229, in _forward
	    generated_sequence = self.model.generate(input_ids=input_ids, attention_mask=attention_mask, **generate_kwargs)
	  File "/data/miniconda3/envs/env-3.8.8/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
	    return func(*args, **kwargs)
	  File "/data/miniconda3/envs/env-3.8.8/lib/python3.8/site-packages/transformers/generation_utils.py", line 1422, in generate
	    return self.sample(
	  File "/data/miniconda3/envs/env-3.8.8/lib/python3.8/site-packages/transformers/generation_utils.py", line 2071, in sample
	    next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
	RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
	```
    In the 4.22 version of transformers, I used the setting of `do_sample=True` and `top_p=0` ​​to make my program do greedy decoding, because the implementation in version 4.22 uses a right shift operation to make sure that there is at least one token left. But when I updated the version to the latest, I got the above exception and took some time to find the reason.

    **The above is the main reason for submitting this PR. I agree that "top_p=0.0 means in theory that no token can be sampled", but in practice, we will encounter a `nan` error with no explicit error message, rather than sample each token with the same odds.**

    (2) In TF and FLAX, the implementation of `TopPLogitsWarper` in both frameworks uses a right-shift operation to achieve top-scoring token preservation, producing the same results as greedy decoding. So using the settings of `do_sample=True` and `top_p=0` will not report an error, which is different from the PyTorch framework. **The behavior of the three frameworks is different now.**

2. When `do_sample=True` and `top_p=1`:
Yes, setting `top_p=1` is equivalent to not having the top_p operation, and it does not cause problems other than significantly reducing generation performance. **My change here is more to keep consistency with the error message of `ValueError`.** 

    Additionally, the developers have ensured that `TopPLogitsWarper` will only be performed if `top_p<1` in the `_get_logits_warper` function of `generation_utils.py`:
 	```
	if top_p is not None and top_p < 1.0:
	    warpers.append(TopPLogitsWarper(top_p=top_p, min_tokens_to_keep=(2 if num_beams > 1 else 1)))
	```
    I just thought it would be better to be consistent.

@NinedayWang thank you for elaborating -- that makes sense! :) 

I'd like to suggest two modifications, to then merge the PR:
1. Regarding making the check on `top_p` more strict: because our library is used in production, we must avoid breaking changes whenever we can (which would be the case). Instead, can we raise a warning when `top_p` is either `0.0` (degenerated to argmax token selection) or `1.0` (redundant operation)?
2. Regarding the case where `top_p=0.0`: we both actually forgot the other argument in the discussion above, `min_tokens_to_keep`, which is not being respected. To fix it in this edge case, changing `if self.min_tokens_to_keep > 1:` to `if self.min_tokens_to_keep > 0:` OR forcing `min_tokens_to_keep` to be strictly positive (and removing this `if`) is enough. This change will also make the edge case return back to a greedy decoding-like behavior, as in TF and FLAX, with no exceptions being thrown.

@gante Thanks for your advice! I will work on this :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@NinedayWang Are you planning to continue this work? :)

> @NinedayWang Are you planning to continue this work? :)

@gante Terribly sorry! Other urgent matters delayed this work. I will continue it and update the progress soon!

@NinedayWang no worries, take your time :D