Sync preprocesses before loading the processor at run_speech_recognition_ctc.py
Author: mpenagar
Pull request: yes

# What does this PR do?

Make sure all processes wait until data is saved before loading the processor from the `output_dir` in the `pytorch/speech-recognition/run_speech_recognition_ctc.py` example.

Issue:
   * Non-main proccesses might try to load the processor from the `output_dir` before it is saved.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.



Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @sanchit-gandhi 

Updated seq2seq ASR fine-tuning script. I'm not very good with github, I guess there is no need to do a new PR.

Wait... there is something I don't get correctly.

As far as I understand from the [(documentation )](https://huggingface.co/transformers/v4.11.3/main_classes/trainer.html#transformers.TrainingArguments.main_process_first) , any code inside a block `with training_args.main_process_first():` should be executed only by the main process:

```
A context manager for torch distributed environment where
on needs to do something on the main process, while blocking
replicas, and when it’s finished releasing the replicas.

One such use is for datasets’s map feature which to be efficient
should be run once on the main process, which upon completion
saves a cached version of results and which then automatically
gets loaded by the replicas.
```

But in my experience, the code is executed by all the processes, not just the main one. Take this minimal `example.py':

```python
from transformers import TrainingArguments,HfArgumentParser
from transformers.trainer_utils import is_main_process

def main():
    parser = HfArgumentParser((TrainingArguments,))
    training_args, = parser.parse_args_into_dataclasses()
    rank = training_args.local_rank
    main_process = is_main_process(rank)
    print(f'\nBEFORE WITH - local_rank={rank} is_main_process={main_process}')
    with training_args.main_process_first():
        print(f'\nINSIDE WITH - local_rank={rank}')

if __name__ == "__main__":
    main()
```

If I execute it in a 4 GPU node:

`OMP_NUM_THREADS=1 python3 -m torch.distributed.launch --nproc_per_node 4 example.py --output_dir none`

The synching is working, but all processes execute the "INSIDE" `print`

Executing with newer `torchrun` does the same:

`OMP_NUM_THREADS=1 torchrun --standalone --nnodes=1 --nproc_per_node=4 example.py --output_dir none`

What I am geting wrong?

No, as the name indicates, it executes the code in the context manager on the main process, and then on all the others. The code is indeed executed in all processes, just in a certain order.

Since with Datasets, everything is cached, executing the preprocessing inside that contextmanager means that process 0 will do the preprocessing, and then all the other will load the result from the cache without needing to do the preprocessing.

Ok, then the PR is not correct, since all the processes will try to write the json files. I removed the original:

`if is_main_process(training_args.local_rank):`

that should be there inside the `with` block...

Indeed, your changes are perfect. Is this ready to be merged now?

It is working in my end without any problem

Is this good for merge @mpenagar? Changes LGTM!

Yes, it is ready. Anyway, I don't know how github works. Should I close the PR (there is a "Close" button there)?

Awesome, thanks for confirming @mpenagar and for your contribution 🤗