[Benchmark] Reuse `optimum-benchmark`
Author: ydshieh
Pull request: yes

# What does this PR do?

[Benchmark] Reuse Otpimum

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30615). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> A naive run gave this:
> 
> ```python
> arthur@brahms ~/transformers (benchmark_reuse_otpimum)> make benchmark                                                                                                                                                                               (py39) 
> python3 benchmark/benchmark.py --config-dir benchmark/config --config-name generation 

@ArthurZucker 

You are using `--commit=diff` but the `main` branch doesn't contain the new files added in this PR. It will only work only if this PR being merged. You can try however with 2 commit sha for now.

No @ydshieh the naive run error is `In 'hydra/config': Could not find 'hydra/job_logging/colorlog'`

Basically we need a `extra["benchmark"]` to install all the packages for us!

> No @ydshieh the naive run error is `In 'hydra/config': Could not find 'hydra/job_logging/colorlog'`

OK, I only saw 

> Additional config directory '/home/arthur/transformers/benchmark/config' not found

sorry.

Anyway, @IlyasMoutawwakil already reduced the dependency and I will remove it.

> but we need to get rid of all the logs and warnings

what do you mean by this? You don't want to see any logs from `optimum-benchmark`?

@ydshieh i think he meant the deprecation warnings of using the `experiment` schema instead of `benchmark` schema.
currently the cli supports both and only warns of their deprecation and how to migrate from one to another.

Hi @IlyasMoutawwakil 

Currently, the way to install `optimum-benchmark` is like 
> pip install optimum-benchmark@git+https://github.com/huggingface/optimum-benchmark.git

Do you think there is a way to add `optimum-benchmark` to `transformers`'s  `setup.py` file?
Otherwise, we can probably add `requirments.txt` in  `transformers`'s `benchmark/`.


Also, @ArthurZucker would like to disable the many logs from `optimum-benchmark` (more precisely, those `[INFO]`).
I  could fine relevant functions like
```
from optimum_benchmark.logging_utils import setup_logging
setup_logging(level="INFO", handlers=["console"])
```
However, since I am running `optimum-benchmark` commandline through a python subprocess, do you think there is a way to disable the log in this case? (maybe it's already controllable via the config file?)

@ydshieh I can do a pypi to make it easier to add to setup.py 🤗 
for logging, I think a `LOG_LEVEL` env var makes sense in this case (can be set in `hydra.job.env_set`).

> `hydra.job.env_set`

Thanks I will give it a try (the logging part).

For installation, I think I will have to update this PR to match the optimum-benchmark recent changes, but it should be somehow minimal I guess. I will wait a pypi from your side the finalize the PR .



Hi @IlyasMoutawwakil I am trying to run with the latest commit `9b28308362702169630ba269f677c11d3d7924b0` (optimum-benchmark). The goal is to match up the new format. However, there is some failure, see below.

Let's work together to have a working commit (if the issue is from there), use it for pypi maybe?


```bash
Traceback (most recent call last):
  File "/usr/local/bin/optimum-benchmark", line 5, in <module>
    from optimum_benchmark.cli import main
  File "/transformers/optimum-benchmark/optimum_benchmark/__init__.py", line 13, in <module>
    from .base import Benchmark
  File "/transformers/optimum-benchmark/optimum_benchmark/base.py", line 12, in <module>
    from .launchers.base import Launcher
  File "/transformers/optimum-benchmark/optimum_benchmark/launchers/base.py", line 10, in <module>
    from .device_isolation_utils import assert_device_isolation
  File "/transformers/optimum-benchmark/optimum_benchmark/launchers/device_isolation_utils.py", line 8, in <module>
    from ..logging_utils import setup_logging
  File "/transformers/optimum-benchmark/optimum_benchmark/logging_utils.py", line 50, in <module>
    def run_subprocess_and_log_stream_output(logger: logging.Logger, args: list[str]) -> Popen:
TypeError: 'type' object is not subscriptable
{}
```

FYI: currently the working version (with the current PR status) is based on `995bf7b035fc6927c673d45f10a7b0b6784b2650` April 30

@ArthurZucker 

I am still waiting for a pypi, but other comments are all addressed.

- no more logs
- run with static/dynamic cache + compile/not compile
etc.

@ydshieh https://pypi.org/project/optimum-benchmark/ 🤗

updated!

Going to merge. Once done,  I will update the example usages (about the commit numbers).

Thanks for the reviews and help (@IlyasMoutawwakil )