might be a waste of resources
Author: SDaoer
Pull request: no

```
        while self._has_unfinished_sequences(this_peer_finished, synced_gpus, device=input_ids.device):
            # prepare model inputs
            model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)

            # forward pass to get next token
            outputs = self(
                **model_inputs,
                return_dict=True,
                output_attentions=output_attentions,
                output_hidden_states=output_hidden_states,
            )

            if synced_gpus and this_peer_finished:
                continue  # don't waste resources running the code we don't need

            ...
```

Why is this condition checked after the `outputs` is generated? Can this be considered a form of resource wastage? Could this part be moved to the beginning of the while loop?
```
if synced_gpus and this_peer_finished:
    continue  # don't waste resources running the code we don't need
```

The code comes from `transformers/generation/utils.py`: `GenerationMixin._sample`

Comments:
cc @gante @zucchini-nlp 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @SDaoer 👋 Thank you for opening this issue 🤗 

TL;DR without it the code will hang in specific settings.

The answer is documented in the code: https://github.com/huggingface/transformers/blob/083e13b7c47f674b11c74d1b7c7ee7cd1241b406/src/transformers/generation/utils.py#L2198

You can complement this comment with the meaning of `synced_gpus`:
```
synced_gpus (`bool`, *optional*):
                Whether to continue running the while loop until max_length. Unless overridden this flag will be set to
                `True` under DeepSpeed ZeRO Stage 3 multiple GPUs environment to avoid hanging if one GPU finished
                generating before other GPUs. Otherwise it'll be set to `False`.
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> Hi @SDaoer 👋 Thank you for opening this issue 🤗
> 
> TL;DR without it the code will hang in specific settings.
> 
> The answer is documented in the code:
> 
> https://github.com/huggingface/transformers/blob/083e13b7c47f674b11c74d1b7c7ee7cd1241b406/src/transformers/generation/utils.py#L2198
> 
> You can complement this comment with the meaning of `synced_gpus`:
> 
> ```
> synced_gpus (`bool`, *optional*):
>                 Whether to continue running the while loop until max_length. Unless overridden this flag will be set to
>                 `True` under DeepSpeed ZeRO Stage 3 multiple GPUs environment to avoid hanging if one GPU finished
>                 generating before other GPUs. Otherwise it'll be set to `False`.
> ```

hi, it appears there has been a misunderstanding in the response provided, as it discusses the rationale behind the necessity of these two lines of code, rather than elucidating the reasons for not positioning them prior to outputs = self(**model_inputs, return_dict=True) to preemptively avert resource wastage. Nonetheless, from a specific standpoint, even if the GPU is freed up at this juncture, it appears that there are no actionable tasks that can be performed with it.