Repository: huggingface/transformers

Issue:
Default process group has not been initialized while using sagemaker data parallel
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.9.0 - dev
- Platform: Sagemaker
- Python version: 
- PyTorch version (GPU?): 1.8.1
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: SQUaD
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run Squad finetune using transformers 4.9.0 - dev

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
```
[1,5]<stdout>:Traceback (most recent call last):
--
[1,5]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
[1,5]<stdout>:    "__main__", mod_spec)
[1,5]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
[1,5]<stdout>:    exec(code, run_globals)
[1,5]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/mpi4py/__main__.py", line 7, in <module>
[1,5]<stdout>:    main()
[1,5]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/mpi4py/run.py", line 196, in main
[1,5]<stdout>:    run_command_line(args)
[1,5]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/mpi4py/run.py", line 47, in run_command_line
[1,5]<stdout>:    run_path(sys.argv[0], run_name='__main__')
[1,5]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 263, in run_path
[1,5]<stdout>:    pkg_name=pkg_name, script_name=fname)
[1,3]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/mpi4py/run.py", line 196, in main
[1,3]<stdout>:    run_command_line(args)
[1,3]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/mpi4py/run.py", line 47, in run_command_line
[1,3]<stdout>:    run_path(sys.argv[0], run_name='__main__')
[1,3]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 263, in run_path
[1,3]<stdout>:    pkg_name=pkg_name, script_name=fname)
[1,3]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 96, in _run_module_code
[1,3]<stdout>:    mod_name, mod_spec, pkg_name, script_name)
[1,3]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
[1,3]<stdout>:    exec(code, run_globals)
[1,3]<stdout>:  File "run_qa.py", line 646, in <module>
[1,3]<stdout>:    main()
[1,3]<stdout>:  File "run_qa.py", line 427, in main
[1,3]<stdout>:    with training_args.main_process_first(desc="train dataset map pre-processing"):
[1,3]<stdout>:  File "/opt/conda/lib/python3.6/contextlib.py", line 81, in __enter__
[1,3]<stdout>:    return next(self.gen)
[1,3]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/transformers/training_args.py", line 1033, in main_process_first
[1,3]<stdout>:    torch.distributed.barrier()
[1,3]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 2419, in barrier
[1,3]<stdout>:    default_pg = _get_default_group()
[1,2]<stdout>:Traceback (most recent call last):
[1,2]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
[1,2]<stdout>:    "__main__", mod_spec)
[1,2]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
[1,2]<stdout>:    exec(code, run_globals)
[1,2]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/mpi4py/__main__.py", line 7, in <module>
[1,2]<stdout>:    main()
[1,2]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/mpi4py/run.py", line 196, in main
[1,2]<stdout>:    run_command_line(args)
[1,2]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/mpi4py/run.py", line 47, in run_command_line
[1,2]<stdout>:    run_path(sys.argv[0], run_name='__main__')
[1,2]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 263, in run_path
[1,2]<stdout>:    pkg_name=pkg_name, script_name=fname)
[1,4]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
[1,4]<stdout>:    "__main__", mod_spec)
[1,4]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
[1,4]<stdout>:    exec(code, run_globals)
[1,4]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/mpi4py/__main__.py", line 7, in <module>
[1,4]<stdout>:    main()
[1,4]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/mpi4py/run.py", line 196, in main
[1,4]<stdout>:    run_command_line(args)
[1,4]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/mpi4py/run.py", line 47, in run_command_line
[1,4]<stdout>:    run_path(sys.argv[0], run_name='__main__')
[1,4]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 263, in run_path
[1,4]<stdout>:    pkg_name=pkg_name, script_name=fname)
[1,4]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 96, in _run_module_code
[1,4]<stdout>:    mod_name, mod_spec, pkg_name, script_name)
[1,4]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
[1,4]<stdout>:    exec(code, run_globals)
[1,4]<stdout>:  File "run_qa.py", line 646, in <module>
[1,4]<stdout>:    main()
[1,4]<stdout>:  File "run_qa.py", line 427, in main
[1,4]<stdout>:    with training_args.main_process_first(desc="train dataset map pre-processing"):
[1,6]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
[1,6]<stdout>:    "__main__", mod_spec)
[1,6]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
[1,6]<stdout>:    exec(code, run_globals)
[1,6]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/mpi4py/__main__.py", line 7, in <module>
[1,6]<stdout>:    main()
[1,6]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/mpi4py/run.py", line 196, in main
[1,6]<stdout>:    run_command_line(args)
[1,6]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/mpi4py/run.py", line 47, in run_command_line
[1,6]<stdout>:    run_path(sys.argv[0], run_name='__main__')
[1,6]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 263, in run_path
[1,6]<stdout>:    pkg_name=pkg_name, script_name=fname)
[1,6]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 96, in _run_module_code
[1,6]<stdout>:    mod_name, mod_spec, pkg_name, script_name)
[1,6]<stdout>:  File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
[1,6]<stdout>:    exec(code, run_globals)
[1,6]<stdout>:  File "run_qa.py", line 646, in <module>
[1,6]<stdout>:    main()
[1,6]<stdout>:  File "run_qa.py", line 427, in main
[1,6]<stdout>:    with training_args.main_process_first(desc="train dataset map pre-processing"):
[1,6]<stdout>:  File "/opt/conda/lib/python3.6/contextlib.py", line 81, in __enter__
[1,6]<stdout>:    return next(self.gen)
[1,6]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/transformers/training_args.py", line 1033, in main_process_first
[1,6]<stdout>:    torch.distributed.barrier()
[1,4]<stdout>:  File "/opt/conda/lib/python3.6/contextlib.py", line 81, in __enter__
[1,4]<stdout>:    return next(self.gen)
[1,4]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/transformers/training_args.py", line 1033, in main_process_first
[1,4]<stdout>:    torch.distributed.barrier()
[1,4]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 2419, in barrier
[1,4]<stdout>:    default_pg = _get_default_group()
[1,4]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 347, in _get_default_group
[1,4]<stdout>:    raise RuntimeError("Default process group has not been initialized, "
[1,4]<stdout>:RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.
```
## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->



Discussion:
@philschmid @sgugger 
From offline discussion, the issue seem to be the following:
1. You can’t use torch.distributed and smdp at the same time. You might want to change torch.distributed.barrier to sm_dist.barrier
2. You could do import either torch.distributed or smdistributed.dataparallel.torch.distributed as dist at the top of the file. Then, you can simply write dist.xyz elsewhere

Likely the PR from which this issue originated is below:
https://github.com/huggingface/transformers/pull/12464


Resolving patch:
diff --git a/src/transformers/training_args.py b/src/transformers/training_args.py
--- a/src/transformers/training_args.py
+++ b/src/transformers/training_args.py
@@ -1052,6 +1052,8 @@ def main_process_first(self, local=True, desc="work"):
                     logger.debug(f"{self.process_index}: waiting for the {main_process_desc} to perform {desc}")
                     if is_torch_tpu_available():
                         xm.rendezvous(desc)
+                    elif is_sagemaker_dp_enabled():
+                        sm_dist.Barrier()
                     else:
                         torch.distributed.barrier()
                 yield
@@ -1061,6 +1063,8 @@ def main_process_first(self, local=True, desc="work"):
                     logger.debug(f"{self.process_index}: {main_process_desc} completed {desc}, releasing all replicas")
                     if is_torch_tpu_available():
                         xm.rendezvous(desc)
+                    elif is_sagemaker_dp_enabled():
+                        sm_dist.Barrier()
                     else:
                         torch.distributed.barrier()
         else:


Test patch: