During the evaluation, the gpu stops, not working
Author: RockMiin
Pull request: no

### System Info

I am fine-tuning Summariation Task with gpt model using multi-gpu. There is no problem training. However, during the evaluation, the following phenomenon occurs. 

The gpu utilization is maintained at 100%, but the temperature is very low.
And the evaluation process is no longer in progress.

How can I solve this problem?

<img width="474" alt="스크린샷 2022-10-27 오후 5 01 10" src="https://user-images.githubusercontent.com/52374789/198227290-ee01b21f-8f1a-4231-8595-546611c96098.png">


### Who can help?

@patil-suraj @patrickvonplaten @LysandreJik 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. I use Seq2Seq TrainingArguements, Trainer
2. I modify prediction_step in Seq2Seq Trainer

```
def prediction_step(
        self,
        model: nn.Module,
        inputs: Dict[str, Union[torch.Tensor, Any]],
        prediction_loss_only: bool,
        ignore_keys: Optional[List[str]] = None,
    ) -> Tuple[Optional[float], Optional[torch.Tensor], Optional[torch.Tensor]]:
        """
        Perform an evaluation step on :obj:`model` using obj:`inputs`.

        Subclass and override to inject custom behavior.

        Args:
            model (:obj:`nn.Module`):
                The model to evaluate.
            inputs (:obj:`Dict[str, Union[torch.Tensor, Any]]`):
                The inputs and targets of the model.

                The dictionary will be unpacked before being fed to the model. Most models expect the targets under the
                argument :obj:`labels`. Check your model's documentation for all accepted arguments.
            prediction_loss_only (:obj:`bool`):
                Whether or not to return the loss only.

        Return:
            Tuple[Optional[float], Optional[torch.Tensor], Optional[torch.Tensor]]: A tuple with the loss, logits and
            labels (each being optional).
        """

        if not self.args.predict_with_generate or prediction_loss_only:
            return super().prediction_step(
                model, inputs, prediction_loss_only=prediction_loss_only, ignore_keys=ignore_keys
            )

        has_labels = "labels" in inputs
        inputs = self._prepare_inputs(inputs)
        if has_labels:
            labels = inputs['labels']
        else:
            labels = None
        generation_inputs = {"input_ids": inputs["input_ids"]}
        slice_start = inputs["input_ids"].shape[-1]
        # XXX: adapt synced_gpus for fairscale as well

        max_length = slice_start + self.generation_max_length if slice_start + self.generation_max_length < 2048 else 2048
        # print(f'num beams : {self.generation_num_beams}')
        gen_kwargs = {
            "max_length": max_length,
            # "min_length": max_length,
            "num_beams": self.generation_num_beams,
            "pad_token_id": self.tokenizer.pad_token_id,
            "eos_token_id": self.tokenizer.eos_token_id,
            "early_stopping": True,
            "synced_gpus": True,
        }

        if self.args.predict_with_generate and not self.args.prediction_loss_only:
            generated_tokens = self.model.generate(
                **generation_inputs,
                **gen_kwargs,
            )
        
            generated_tokens = generated_tokens[:, slice_start:]
            
            if generated_tokens.shape[-1] < gen_kwargs["max_length"]:
                generated_tokens = self._pad_tensors_to_max_len(generated_tokens, gen_kwargs["max_length"])
        

        with torch.no_grad():
            with self.autocast_smart_context_manager():
                outputs = model(input_ids=inputs['input_ids'], labels=inputs['input_ids'])
            if has_labels:
                if self.label_smoother is not None:
                    loss = self.label_smoother(outputs, inputs["input_ids"])
                else:
                    loss = (outputs["loss"] if isinstance(outputs, dict) else outputs[0])
            else:
                loss = None
        loss = None
        if self.args.prediction_loss_only:
            return (loss, None, None)

        return (loss, generated_tokens, labels)
```

### Expected behavior

I also tried fine-tuning the translation task
However, there was no error in the translation task, and it seems to occur when the length of the sentence generated is long from experience
It has been tried several times, but it has occurred at different points, and it has been confirmed that it is not a data problem.

I'd like your help.
Thank you

Comments:
It's possible there are tensors not all of the same lengths across processes (maybe the labels since they are not padded?). When trying to gather them, torch.distributed just hangs instead of throwing an error.

@sgugger 
First of all, thank you for your answer.
The label is padded.
When generating, it seems to be a problem that occurs because the end point of the sentence is different for each gpu process.
How can we solve this?

@sgugger Can you help me about this issue?
Thank you

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.