Repository: huggingface/accelerate

Issue:
cannot import accelerate when torch>=2.0.1 and torch.distributed is disabled
### System Info

```Shell
I can't run `accelerate env` because of an import error.

accelerate: 0.21.0
OS: macOS
python: 3.10.12
numpy: 1.24.2
torch: 2.0.1
```


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] One of the scripts in the examples/ folder of Accelerate or an officially supported `no_trainer` script in the `examples` folder of the `transformers` repo (such as `run_no_trainer_glue.py`)
- [ ] My own task or dataset (give details below)

### Reproduction

1. build `torch >= 2.0.1` with `USE_DISTRIBUTED=0`
2. install `accelerate == 0.21.0`
3. `python -c "import accelerate"`
4. raise `ModuleNotFoundError: No module named 'torch._C._distributed_c10d'; 'torch._C' is not a package`

<details><summary>Traceback (most recent call last):</summary><div>

```
  File "<string>", line 1, in <module>
  File "/nix/store/v9h5iiawvw6y0j03840qxjpqc9nbk4c2-python3-3.10.12-env/lib/python3.10/site-packages/accelerate/__init__.py", line 3, in <module>
    from .accelerator import Accelerator
  File "/nix/store/v9h5iiawvw6y0j03840qxjpqc9nbk4c2-python3-3.10.12-env/lib/python3.10/site-packages/accelerate/accelerator.py", line 35, in <module>
    from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state
  File "/nix/store/v9h5iiawvw6y0j03840qxjpqc9nbk4c2-python3-3.10.12-env/lib/python3.10/site-packages/accelerate/checkpointing.py", line 24, in <module>
    from .utils import (
  File "/nix/store/v9h5iiawvw6y0j03840qxjpqc9nbk4c2-python3-3.10.12-env/lib/python3.10/site-packages/accelerate/utils/__init__.py", line 132, in <module>
    from .fsdp_utils import load_fsdp_model, load_fsdp_optimizer, save_fsdp_model, save_fsdp_optimizer
  File "/nix/store/v9h5iiawvw6y0j03840qxjpqc9nbk4c2-python3-3.10.12-env/lib/python3.10/site-packages/accelerate/utils/fsdp_utils.py", line 24, in <module>
    import torch.distributed.checkpoint as dist_cp
  File "/nix/store/v9h5iiawvw6y0j03840qxjpqc9nbk4c2-python3-3.10.12-env/lib/python3.10/site-packages/torch/distributed/checkpoint/__init__.py", line 1, in <module>
    from .metadata import (
  File "/nix/store/v9h5iiawvw6y0j03840qxjpqc9nbk4c2-python3-3.10.12-env/lib/python3.10/site-packages/torch/distributed/checkpoint/metadata.py", line 3, in <module>
    from torch.distributed._shard.sharded_tensor.metadata import TensorProperties
  File "/nix/store/v9h5iiawvw6y0j03840qxjpqc9nbk4c2-python3-3.10.12-env/lib/python3.10/site-packages/torch/distributed/_shard/__init__.py", line 1, in <module>
    from .api import (
  File "/nix/store/v9h5iiawvw6y0j03840qxjpqc9nbk4c2-python3-3.10.12-env/lib/python3.10/site-packages/torch/distributed/_shard/api.py", line 5, in <module>
    from torch.distributed import distributed_c10d
  File "/nix/store/v9h5iiawvw6y0j03840qxjpqc9nbk4c2-python3-3.10.12-env/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 16, in <module>
    from torch._C._distributed_c10d import (
```
</div></details>

### Expected behavior

This is the line in the issue.
https://github.com/huggingface/accelerate/blob/d5894ab4992878c191ed4c9fdb53b35ad2c0e628/src/accelerate/utils/fsdp_utils.py#L23-L24

I think it would be better to decide whether to import `torch.distributed` by the result of `torch.distributed.is_available()` besides the torch version.


Discussion:
Yes, we need a `torch.distributed.is_available()` in that test in case PyTorch was built without distributed support, cc @pacman100 

Resolving patch:
diff --git a/src/accelerate/utils/other.py b/src/accelerate/utils/other.py
--- a/src/accelerate/utils/other.py
+++ b/src/accelerate/utils/other.py
@@ -28,7 +28,7 @@
 from ..state import PartialState
 from .constants import FSDP_PYTORCH_VERSION
 from .dataclasses import DistributedType
-from .imports import is_deepspeed_available, is_safetensors_available, is_tpu_available
+from .imports import is_deepspeed_available, is_safetensors_available, is_torch_distributed_available, is_tpu_available
 from .transformer_engine import convert_model
 from .versions import is_torch_version
 
@@ -77,7 +77,7 @@ def extract_model_from_parallel(model, keep_fp32_wrapper: bool = True):
 
         options += (DeepSpeedEngine,)
 
-    if is_torch_version(">=", FSDP_PYTORCH_VERSION):
+    if is_torch_version(">=", FSDP_PYTORCH_VERSION) and is_torch_distributed_available():
         from torch.distributed.fsdp.fully_sharded_data_parallel import FullyShardedDataParallel as FSDP
 
         options += (FSDP,)


Test patch: