Undocumented distributed inference behaviour for `run_summarization.py`
Author: alexcoca
Pull request: no
Labels: name: WIP

name: bug

### System Info

```shell
Fails with error


Traceback (most recent call last):
  File "/scratches/neuron/anaconda3/envs/T5DST-SGD/bin/transformers-cli", line 5, in <module>
    from transformers.commands.transformers_cli import main
  File "/scratches/neuron/anaconda3/envs/T5DST-SGD/lib/python3.8/site-packages/transformers/commands/transformers_cli.py", line 26, in <module>
    from .user import UserCommands
  File "/scratches/neuron/anaconda3/envs/T5DST-SGD/lib/python3.8/site-packages/transformers/commands/user.py", line 20, in <module>
    from huggingface_hub.hf_api import HfFolder, create_repo, list_repos_objs, login, logout, whoami
ImportError: cannot import name 'list_repos_objs' from 'huggingface_hub.hf_api' (/scratches/neuron/anaconda3/envs/T5DST-SGD/lib/python3.8/site-packages/huggingface_hub/hf_api.py)


However I am running `4.16.2` with python `3.8`.
```


### Who can help?

@sgugger @stevhliu @patil-suraj 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I am working with a copy of the `run_summarization.py` (pytorch) example that the authors of this [paper](https://arxiv.org/pdf/2109.07506.pdf) modified to work for dialogue state tracking (implemented [here ](https://github.com/chiahsuan156/DST-as-Prompting) for reference)

The `run_summarization.py` script can be launched with `torch.distributed.launch` and the `--do_predict` option. This shards the examples in the test set to various GPUs and therefore generation and task-oriented metrics is accelerated. The predictions are written to the `generated_predictions.txt` file in the output directory. 

To be able to compute dialogue-relevant task oriented metrics, one ought to run a postprocessing script that uses the `generated_predictions.txt`. Because the trainer erases all the columns that are not keys to the model `forward` method from the dataset, the metadata that informs us of what training examples the predictions are related to is lost. Therefore, we rely on the ordering of the `generated_predictions.txt` to match the order of the examples in the dataset.

My question is: 
 - Does `predictions` (`L675`)  obey the order of the `dataset`? So if my dataset has 1m examples, will the 1m entries in the `predictions` list match the order of the dataset iterator? In my experience this depends on implementation* and the behaviour is not documented.
 
*For example, in frameworks such as `ray` you have to explicitly enforce the order in which the results are returned and the predictions may be returned out of order - if a process finishes, it returns its results so it can be given more work by an external load balancer.

### Expected behavior

```shell
Improved documentation about expected behaviour here. Happy to discuss where this should be added and contribute a small PR to clarify this important issue.
```


Comments:
I'm not sure where the bug is. It sounds like you have a question, which should be asked on the [forums](https://discuss.huggingface.co/).

`Trainer.predict` will return predictions in the same order as the underlying dataset, regardless of the setup you're using to get your predictions. I don't feel it requires documentation as it's the intended behavior of the method. There would be a warning if it was not the case.

Hi @sgugger, I apologise for raising this issue. I was expecting to find info about async behaviour in docs and did not.

I just finished reading the code and I see that `L174` in `trainer_pt_utils.py` calls `torch.distributed.all_gather` with `async_op=False` so we preserve the order as you say.

Do you think it would be worth expanding the docs? We could  show how to run distributed inference for `run_summarization.py` or add one sentence in the part of the docs that tells us how to run distributed training with a short paragraph on how to do distributed inference with a note "the order of the underlying dataset will be preserved"?

If you don't think this adds value, feel free to close the issue straight away.

Thank you for your helpful answer.

If you feel the doc needs to be expanded, I'm happy to review a PR, I just told you why I thought it wasn't worth mentioning when I wrote the current doc of `Trainer.predict` ;-)
But adding some lines on how to run distributed inference are more than welcome!

Ok, I'll add that to a TODO as this should be a quick one. Let's label this as WIP as I expect this would be a couple of weeks given my current workload.

Just an update - I did manage to successfully deploy training code written with the `Trainer` API on a `SLURM` cluster using `torchrun` ([here](https://pytorch.org/docs/stable/elastic/run.html?highlight=torchrun)). We can discuss where in the docs it would be best to show an example - I think it would help a lot of people. There are some posts on the forum I can update, to start with.