Allow users to pass through `Accelerate` instance
Author: nialldevlin1
Pull request: no
Labels: name: Feature request

### Feature request

Users can create an `Accelerator` instance and optionally pass this through to the `Trainer`, which should then handle further postprocessing based on the state of the object.

### Motivation

In the base `Trainer` class, the call to `create_accelerator_and_postprocess` creates the `Accelerator` instance and then sets flags detailing whether certain features/modes of operation are enabled. Within this method, a check for the current accelerate version is made. If the version is greater than 0.28 then only the dataloader args are added to the arguments passed through to the `Accelerate` instantiation:

```python
args = {
    "deepspeed_plugin": self.args.deepspeed_plugin,
}
if is_accelerate_available("0.28.0"):
    args["dataloader_config"] = dataloader_config
else:
    args.update(accelerator_config)
```

This prevents any custom setup that might be required. It is also not clear what parts of the config are actually applied and isn't helped by the fact that accelerate itself uses arguments and environment variables to silently set things - allowing a means for the user to explicitly set the state of the `Accelerate` instance is a much clearer pattern.

Instead, users should be able to create their own `Accelerate` argument, pass it through to the `Trainer`, which should then continue with the postprocessing, e.g.:

```python
def maybe_create_accelerator_and_postprocess(self, accelerator: Optional[Accelerator] = None):

    self.accelerator = accelerator or self.create_accelerator()

    self.gather_function = self.accelerator.gather_for_metrics

    if "use_gather_object" in inspect.signature(self.gather_function).parameters:
        self.gather_function = functools.partial(
            self.gather_function, use_gather_object=self.args.eval_use_gather_object
        )
    ...
```

### Your contribution

Can offer assistance, although I imagine there are several gotchas in the library that preclude certain setups that I'm not aware of

Comments:
cc @SunMarc 

Hi, I have raised a PR to add the above discussed functionality. Requesting you to please have a look and let me know further. Thanks!

@SunMarc requesting your review, please let me know in case any changes/corrections are needed

`TrainingArguments` has `accelerator_config`, `AcceleratorConfig` has field `use_configured_state`. Could that be safely used to reuse the existing state from an `Accelerator` object created **before** `SFTConfig` or `SFTTrainer`?

In case that's helpful and for context, here's pseudo code of my use-case:

```py
def main(...):
  # programmatically configure accelerator
  fsdp_plugin = ...
  accelerator = Accelerator(fsdp_plugin=...)

  # do something on main first as an example of something that requires an accelerator object
  with accelerator.main_process_first():
    ...

  # configure/run SFT using configuration from the accelerator object above
  training_args = SFTConfig(accelerator_config=AcceleratorConfig(use_configured_state=True), ...)
  trainer = SFTTrainer(args=training_args, ...)
  trainer.train()
```

A quick follow up, some links worth taking a look at:
 * https://github.com/huggingface/transformers/issues/40376
 * https://github.com/huggingface/transformers/pull/29781

@nialldevlin1 I suspect given https://github.com/huggingface/transformers/issues/40376#issuecomment-3223895735, this issue can probably be closed (as duplicate).

 Thanks @ravwojdyla, I wasn't aware of this, does this new `use_configured_state` work fine?

You should indeed use `use_configured_state` ! 

> does this new use_configured_state work fine?

@Flakes342 yep it worked fine



@nialldevlin1 https://github.com/huggingface/transformers/issues/40616#issuecomment-3326533877. This issue can be closed now I guess