log_softmax runtimeerror when utilizing generation gradients in beam_search 
Author: pedrocolon93
Pull request: no

## Environment info

- `transformers` version: 4.11.0 (also in latest)
- Platform: Linux-5.4.0-88-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.9.1+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No (but also happens in gpu)
- Using distributed or parallel set-up in script?: N/A

### Who can help
@patrickvonplaten

## Information

Model I am using BART:

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```python
#Train the generator
                batch_size = batch["input_ids"].shape[0]

                length_penalty = 1.0
                early_stopping = True
                num_beams = 5

                beam_scorer = BeamSearchScorer(
                    batch_size=batch_size,
                    max_length=self.max_target_length,
                    num_beams=num_beams,
                    device=self.generator.device,
                    length_penalty=length_penalty,
                    do_early_stopping=early_stopping,
                    num_beam_hyps_to_keep=1,
                )
                # interleave with `num_beams`
                input_ids = torch.ones((num_beams * batch_size, 1), device=self.generator.device, dtype=torch.long)
                input_ids = input_ids * self.generator.model.config.decoder_start_token_id
                encoder_outputs = self.generator.model.get_encoder()(
                    batch['input_ids'].repeat_interleave(num_beams, dim=0),
                    return_dict=True)
                logits_processor = LogitsProcessorList([
                    MinLengthLogitsProcessor(5, eos_token_id=self.generator.model.config.eos_token_id),
                ])

                fake = self.generator.model.beam_search(
                    input_ids,
                    beam_scorer,
                    logits_processor=logits_processor,
                    max_length=self.max_target_length,
                    pad_token_id=self.generator.tokenizer.pad_token_id,
                    eos_token_id=self.generator.tokenizer.eos_token_id,
                    output_scores=True,
                    return_dict_in_generate=True,
                    encoder_outputs=encoder_outputs
                )
```

1. Essentially, when utilizing the "scores" that beam search produces (after filtering out the ones used in the returned sequence), eventually if you utilize some kind of loss with those scores vectors (i.e. multiplying the scores with an embedding matrix and summing it up) like passing that on to some model that can generate a loss, you will get the error:
 `RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [10, 50296]], which is output 0 of LogSoftmaxBackward, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
`
2. With debugging, you can figure out that the error can be traced back to the use of log_softmax in generation_utils: 
- https://huggingface.co/transformers/_modules/transformers/generation_utils.html
```python
 next_token_scores = **nn.functional.log_softmax(**
                next_token_logits, dim=-1
            )  # (batch_size * num_beams, vocab_size)
```
Now, I don't think this is an issue with transformers (i.e. it should just work given that its a pytorch call), but it is blocking me. My solution (once more sloppy) was to replace it with an actual log(softmax( call:
```python
            next_token_scores = torch.log(F.softmax(next_token_logits, dim=-1))  # (batch_size * num_beams, vocab_size)
```

## Expected behavior

Utilizing the gradients from the beam search scores (at every generation point) should work for backpropagation and other custom uses. 


Comments:
Small clarification: log_softmax is numerically stable, whereas log(softmax( is not, so I understand this is not ideal. 

Thanks for the issue @pedrocolon93 - we don't really support gradient backpropagation with `generate(...)` yet. I don't think we can just replace `log_softmax` with `torch.log(F.softmax(...))` as the function is not numerical stable as you pointed out. 

`log_softmax(...)` should also be able to compute the gradient correctly...the issue IMO rather comes from using `logits_processor` that perform an in-place operation no? Can you try whether the gradients work with `log_softmax(...)` and no `logits_processor`?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.