cannot import name 'ReduceOp' from 'torch.distributed'
Author: HugeBob
Pull request: no

### System Info

Transformers version: 4.21.2
Platform: NVIDIA Jetson Xavier NX
Python version: 3.8.10
PyTorch version: '1.13.0a0+936e9305.nv22.11'

Errors out when trying to import SegformerForSemanticSegmentation, came back to test my code after not using it for a long while, changed nothing about the environment (didn't even turn the machine on) and it no longer works. Been trying to figure out what the problem is but finally gave up and decided to check if anyone else was having the problem, didn't see anything so I'm sure I'm just doing something stupid.

`from transformers import SegformerForSemanticSegmentation`

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 1002, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/segformer/modeling_segformer.py", line 28, in <module>
    from ...modeling_utils import PreTrainedModel
  File "/usr/local/lib/python3.8/dist-packages/transformers/modeling_utils.py", line 79, in <module>
    from accelerate import dispatch_model, infer_auto_device_map, init_empty_weights
  File "/usr/local/lib/python3.8/dist-packages/accelerate/__init__.py", line 7, in <module>
    from .accelerator import Accelerator
  File "/usr/local/lib/python3.8/dist-packages/accelerate/accelerator.py", line 27, in <module>
    from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state
  File "/usr/local/lib/python3.8/dist-packages/accelerate/checkpointing.py", line 24, in <module>
    from .utils import (
  File "/usr/local/lib/python3.8/dist-packages/accelerate/utils/__init__.py", line 68, in <module>
    from .operations import (
  File "/usr/local/lib/python3.8/dist-packages/accelerate/utils/operations.py", line 25, in <module>
    from torch.distributed import ReduceOp
ImportError: cannot import name 'ReduceOp' from 'torch.distributed' (/usr/local/lib/python3.8/dist-packages/torch/distributed/__init__.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1039, in _handle_fromlist
  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 993, in __getattr__
    value = getattr(module, name)
  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 992, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 1004, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.models.segformer.modeling_segformer because of the following error (look up to see its traceback):
cannot import name 'ReduceOp' from 'torch.distributed' (/usr/local/lib/python3.8/dist-packages/torch/distributed/__init__.py)
```

### Who can help?

@LysandreJik SegformerForSemanticSegmentation

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

`from transformers import SegformerForSemanticSegmentation`

### Expected behavior

I would expect to be able to import SegformerForSemanticSegmentation and use the class to load my already existing Segformer model. My script used to work but now errors out at the import after not touching the machine for at least a month (didn't even turn it on)

Comments:
Changed from using SegformerForSemanticSegmentation to using AutoModelForSemanticSegmentation, the import now works fine but loading the pretrained model does not.

```
from transformers import AutoFeatureExtractor, AutoModelForSemanticSegmentation
feature_extractor = AutoFeatureExtractor.from_pretrained("nvidia/segformer-b4-finetuned-ade-512-512")
segment_model = AutoModelForSemanticSegmentation.from_pretrained("segments-tobias/segformer-b0-finetuned-segments-sidewalk")
```

Same error, stack trace pasted below

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 1002, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/segformer/modeling_segformer.py", line 28, in <module>
    from ...modeling_utils import PreTrainedModel
  File "/usr/local/lib/python3.8/dist-packages/transformers/modeling_utils.py", line 79, in <module>
    from accelerate import dispatch_model, infer_auto_device_map, init_empty_weights
  File "/usr/local/lib/python3.8/dist-packages/accelerate/__init__.py", line 7, in <module>
    from .accelerator import Accelerator
  File "/usr/local/lib/python3.8/dist-packages/accelerate/accelerator.py", line 27, in <module>
    from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state
  File "/usr/local/lib/python3.8/dist-packages/accelerate/checkpointing.py", line 24, in <module>
    from .utils import (
  File "/usr/local/lib/python3.8/dist-packages/accelerate/utils/__init__.py", line 68, in <module>
    from .operations import (
  File "/usr/local/lib/python3.8/dist-packages/accelerate/utils/operations.py", line 25, in <module>
    from torch.distributed import ReduceOp
ImportError: cannot import name 'ReduceOp' from 'torch.distributed' (/usr/local/lib/python3.8/dist-packages/torch/distributed/__init__.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py", line 445, in from_pretrained
    model_class = _get_model_class(config, cls._model_mapping)
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py", line 359, in _get_model_class
    supported_models = model_mapping[type(config)]
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py", line 564, in __getitem__
    return self._load_attr_from_module(model_type, model_name)
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py", line 578, in _load_attr_from_module
    return getattribute_from_module(self._modules[module_name], attr)
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py", line 534, in getattribute_from_module
    if hasattr(module, attr):
  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 992, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py", line 1004, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.models.segformer.modeling_segformer because of the following error (look up to see its traceback):
cannot import name 'ReduceOp' from 'torch.distributed' (/usr/local/lib/python3.8/dist-packages/torch/distributed/__init__.py)
```

Solved by upgrading the PyTorch version 1.13.0, had to build from source with USE_DISTRIBUTED=1

You ever think a problem is solved but then you later figure out that your problem actually isn't solved? Yea I didn't get the same error as last time because the new PyTorch build didn't have CUDA enabled, when using a proper CUDA enabled PyTorch install I do still get this error. Gonna drop the ping once more as this likely would have gotten lost (sorry for the mess) @LysandreJik 

Reverting to PyTorch 1.11.0 resolved this problem but gives the following warning:

```
/usr/local/lib/python3.8/dist-packages/transformers/models/segformer/image_processing_segformer.py:102: FutureWarning: The `reduce_labels` parameter is deprecated and will be removed in a future version. Please use `do_reduce_labels` instead.
```

I presume the Segformer model in Transformers simply relies on a portion of PyTorch that was deprecated starting at PyTorch version 1.12.0?

No this deprecation comes from the Transformers library, you should use the argument indicated.

But the error happens just when trying to initialize a Segformer model, would the solution be to update Transformers?

Error persists on Transformers 4.25.1

It's not an error, just a warning or are you still having the original issue?

I get the original issue on PyTorch 1.12.0+, it works fine on PyTorch 1.11.0

Any idea if the SegformerForSemanticSegmentation will be updated to support PyTorch 1.12 or has it been abandoned?

There is nothing wrong with segformer, the problem stems from your PyTorch install. We have tried to reproduce your issue with @muellerzr but with all versions of PyTorch from 1.11 to 1.13 there is nothing wrong with the import that fails on your setup.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I still have the same issue with :
pyTorch : 2.0.0+cpu
python : 3.9.1
transformers : 4.25.1

I found this issue by trying this clothes segmentation :
https://huggingface.co/mattmdjaga/segformer_b2_clothes

Error :

```
/lib/python3.9/site-packages/transformers/models/segformer/image_processing_segformer.py:102: FutureWarning: The `reduce_labels` parameter is deprecated and will be removed in a future version. Please use `do_reduce_labels` instead.
  warnings.warn(
```

Okay it is a warning, but the pipe seems broken at the end.