Wandb hyperparameter search: Please make sure to properly initialize your accelerator via `accelerator = Accelerator()`
Author: BramVanroy
Pull request: no

### System Info

- `transformers` version: 4.32.1
- Platform: Linux-5.14.0-284.25.1.el9_2.x86_64-x86_64-with-glibc2.34
- Python version: 3.11.4
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.22.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)

### Who can help?

@muellerzr and @pacman100


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When I am using hyperparameter search as described [here](https://huggingface.co/docs/transformers/hpo_train) with wandb, I am getting this error 

> Please make sure to properly initialize your accelerator via `accelerator = Accelerator()` before using any functionality from the `accelerate` library.

This only happens when using hyperparameter search and not in other cases where I use the trainer for training. So I think this is an issue with the hyperparameter search functionality and not with accelerate specifically.

The error above occurs when I use `torchrun` in a single node with four GPUs. For debugging, I tried running it on a single GPU. That leads to another error:

> AttributeError("'AcceleratorState' object has no attribute 'distributed_type'")

Unfortunately wandb is not giving me the full error trace but there's only a [handful of places](https://github.com/search?q=repo%3Ahuggingface%2Ftransformers%20distributed_type&type=code) where `distrubted_type` occurs, so someone with good knowledge of the accelerate integration in the trainer would probably know where to look..

### Expected behavior

No crash. :-)

Comments:
@BramVanroy BTW, saw this nit if you weren't aware at the bottom of that doc:

> Currently, Hyperparameter search for DDP is enabled for optuna and sigopt. Only the rank-zero process will generate the search trial and pass the argument to other ranks.

(Though I'm exploring/trying this today, bare minimum for the non-distributed case)

Also, I could get it working on a single GPU, going to test multi-GPU in a moment. Build from the `main` branch (as some of it is due to the training arguments, but a totally different error to you), and try running the following via `python {my_script.py}`:

```python
from datasets import load_dataset
from transformers import AutoTokenizer, DataCollatorWithPadding, TrainingArguments, AutoModelForSequenceClassification, Trainer

raw_datasets = load_dataset("glue", "mrpc")
checkpoint = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)


def tokenize_function(example):
    return tokenizer(example["sentence1"], example["sentence2"], truncation=True)

def wandb_hp_space(trial):
    return {
        "method": "random",
        "metric": {"name": "objective", "goal": "minimize"},
        "parameters": {
            "learning_rate": {"distribution": "uniform", "min": 1e-6, "max": 1e-4},
            "per_device_train_batch_size": {"values": [16, 32, 64, 128]},
        },
    }


tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)
data_collator = DataCollatorWithPadding(tokenizer=tokenizer)
training_args = TrainingArguments("test-trainer")

def model_init(trial):
    return AutoModelForSequenceClassification.from_pretrained(
        checkpoint, 
        num_labels=2
    )
trainer = Trainer(
    model=None,
    args=training_args,
    model_init=model_init,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["validation"],
    data_collator=data_collator,
    tokenizer=tokenizer,
)

best_trial = trainer.hyperparameter_search(
    direction="maximize",
    backend="wandb",
    hp_space=wandb_hp_space,
    n_trials=2,
)
```

I also receive a non-accelerate related error, which seems to stem from it being unsupported on multi-GPU, so the problem here should be solved. Can you try building from that branch again, and not running with torchrun? (or use a different logging method, if so)

Hey @muellerzr sorry for getting back to you so late. Updating from 4.32.1 to 4.33.1 seems to have solved the issue indeed. Thanks a lot!