Allowing one to pass run_config to hyperparameter tuning (to allow storing checkpoints on s3)
Author: hugocool
Pull request: no

### Feature request

[in ray 2.5.1 it becomes possible to set the intermediate storage of checkpoints to s3](https://docs.ray.io/en/latest/tune/tutorials/tune-storage.html#configuring-tune-with-cloud-storage-aws-s3-google-cloud-storage).
```
    from ray.air.config import RunConfig
    from ray.air.integrations.mlflow import MLflowLoggerCallback


    run_config = RunConfig(
        storage_path="s3://.....",
        callbacks=[MLflowLoggerCallback],
    )
```
However in the huggingface integration it is not possible to pass this kwarg to the tuner, like so:
```
            best_run = trainer.hyperparameter_search(
                direction="maximize",
                backend="ray",
                run_config=run_config,
            )  
```

### Motivation

to tune a transformer without clogging local storage

### Your contribution

i could submit a proposal

Comments:
Hi @hugocool, 

Thanks for raising this issue. The integrations are maintained by their authors and not us. You can definitely open a PR, just make sure to ping them to verify the change! 
 

`transformers.integrations.run_hp_search_ray` calls `ray.tune.run` which doesn't accept `run_config` as that is part of the newer `Tuner` API. According to https://discuss.ray.io/t/tune-run-vs-tuner-fit/7041/3 `Tuner` is supposed to replace `run` in the long term, although for now it's still beta. `transformers` should probably migrate to `Tuner` at some point, I don't know if/when that'll be a good idea.

In the meantime, `ray.tune.run` directly accepts `callbacks` and `storage_path` arguments, so I think passing them to `hyperparameter_search` without wrapping in a `RunConfig` should just work?

That’s correct, somewhat hidden in ray.tune one finds that **kwargs are passed, unfortunately it is not documented, nor what settings to use in order to prevent local accumulation of checkpoints to the point your disk fills (which is governed by the `check_point_freq`).

So maybe on update to the documentation is in order?
I could provide an example of how to run HF tuning on AWS batch for example?


On 20 Jun 2023 at 18:14 +0200, Alex Hall ***@***.***>, wrote:
> transformers.integrations.run_hp_search_ray calls ray.tune.run which doesn't accept run_config as that is part of the newer Tuner API. According to https://discuss.ray.io/t/tune-run-vs-tuner-fit/7041/3 Tuner is supposed to replace run in the long term, although for now it's still beta. transformers should probably migrate to Tuner at some point, I don't know if/when that'll be a good idea.
> In the meantime, ray.tune.run directly accepts callbacks and storage_path arguments, so I think passing them to hyperparameter_search without wrapping in a RunConfig should just work?
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you were mentioned.Message ID: ***@***.***>


> somewhat hidden in ray.tune one finds that **kwargs are passed

Do you mean hidden in `transformers.hyperparameter_search`/`run_hp_search_ray`?

> I could provide an example of how to run HF tuning on AWS batch for example?

I don't know who you're offering this to. I'm not using this myself, I'm looking for ways to contribute to this repo in general.

> Do you mean hidden in transformers.hyperparameter_search/run_hp_search_ray?
Yes, there and within `ray/tune/tune.py`; `run`
> > I could provide an example of how to run HF tuning on AWS batch for example?
> > I don't know who you're offering this to. I'm not using this myself, I'm looking for ways to contribute to this repo in general.
I’m sorry for being a little vague, basically the documentation is lacking, I had to dig through GitHub issues, forum posts and source code to figure out how to do this.
The API that hugging face exposes is deceptively simple, it seems on the surface like it will just work while in reality this is not the case.
This is not helped by the Enum choice connection to the where you provide backend=‘ray’, and now type checking doesn’t help you..
So, more documentation could definitely help people I think.

Maybe I could help provide more documentation? Based on the examples I worked out?
Idk, id hope that would help others!


___

Hugo Evers
On 20 Jun 2023 at 18:42 +0200, Alex Hall ***@***.***>, wrote:
> > somewhat hidden in ray.tune one finds that **kwargs are passed
> Do you mean hidden in transformers.hyperparameter_search/run_hp_search_ray?
> > I could provide an example of how to run HF tuning on AWS batch for example?
> I don't know who you're offering this to. I'm not using this myself, I'm looking for ways to contribute to this repo in general.
> —
> Reply to this email directly, view it on GitHub, or unsubscribe.
> You are receiving this because you were mentioned.Message ID: ***@***.***>


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.