Regression: Need to guard torch.distributed.is_initialized with torch.distributed.is_available
Author: marr75
Pull request: no

### System Info

- `transformers` version: 4.33.1
- Platform: macOS-13.5-arm64-arm-64bit
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

Accessing via sentence-transformers so text models? @ArthurZucker and @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

My environment.yaml:

name: prompt_gym

channels:
  - conda-forge
  - pytorch

dependencies:
  - faiss
  - python=3.10
  - sentence-transformers

1. `mamba create` from the same directory as the environment.yaml
2. `mamba activate prompt_gym`
3.  From a python interpreter:
```python
import sentence_transformers
encoder = sentence_transformers.SentenceTransformer('intfloat/e5-small-v2', device='mps')
```
4. Receive error:
```text
  File "~/Documents/dev/gpt-experiments/common/src/msw_gpt_common/embedding.py", line 47, in encoder
    encoder = sentence_transformers.SentenceTransformer("intfloat/e5-base-v2", device="mps")
  File "~/mambaforge/envs/prompt_gym/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 95, in __init__
    modules = self._load_sbert_model(model_path)
  File "~/mambaforge/envs/prompt_gym/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 840, in _load_sbert_model
    module = module_class.load(os.path.join(model_path, module_config['path']))
  File "~/mambaforge/envs/prompt_gym/lib/python3.10/site-packages/sentence_transformers/models/Transformer.py", line 137, in load
    return Transformer(model_name_or_path=input_path, **config)
  File "~/mambaforge/envs/prompt_gym/lib/python3.10/site-packages/sentence_transformers/models/Transformer.py", line 29, in __init__
    self._load_model(model_name_or_path, config, cache_dir)
  File "~/mambaforge/envs/prompt_gym/lib/python3.10/site-packages/sentence_transformers/models/Transformer.py", line 49, in _load_model
    self.auto_model = AutoModel.from_pretrained(model_name_or_path, config=config, cache_dir=cache_dir)
  File "~/mambaforge/envs/prompt_gym/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 563, in from_pretrained
    return model_class.from_pretrained(
  File "~/mambaforge/envs/prompt_gym/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2347, in from_pretrained
    if is_fsdp_enabled():
  File "~/mambaforge/envs/prompt_gym/lib/python3.10/site-packages/transformers/modeling_utils.py", line 118, in is_fsdp_enabled
    return torch.distributed.is_initialized() and strtobool(os.environ.get("ACCELERATE_USE_FSDP", "False")) == 1
AttributeError: module 'torch.distributed' has no attribute 'is_initialized'
```

### Expected behavior

Transformers should not throw an error when loading an Automodel from a pretrained model. It should check that distributed processing is available before checking if it is initialized. See #17590 for an earlier version of the same bug. Regression was likely introduced by #25686.

Comments:
cc @pacman100 

I am using FSDP to train 70B models according to your blog post @pacman100 . I found that https://github.com/huggingface/transformers/commit/73b13ac099443cf8297a4b729f00c907fa00f1b5 you add a `torch.distributed.is_initialized()` check. I found it is `False` when I use `accelerate launch`.

Could you please give me some suggestions?

```python
def is_fsdp_enabled():
    return (
        torch.distributed.is_available()
        # and torch.distributed.is_initialized()
        and strtobool(os.environ.get("ACCELERATE_USE_FSDP", "False")) == 1
    )
```

I comment this line to make FSDP load large model works.