Fix W&B run name
Author: qubvel
Pull request: yes

# What does this PR do?

Currently, the `TrainingArguments.run_name` argument is **silently** ignored for W&B if it is the same as `TrainingArguments.output_dir`. 
https://github.com/huggingface/transformers/blob/d1d94d798f1ed5c0b5de9a794381aeb7dc319c12/src/transformers/integrations/integration_utils.py#L766

This behavior was introduced in this [PR](https://github.com/huggingface/transformers/pull/21035) to avoid runs with the same name in W&B. However, 
1) we can still have duplicate run names if their name is not equal to `output_dir`
2) silent ignoring of an argument may lead to a feeling that the argument is just not working

In my opinion, runs with the same name are normal behavior, they still can be differentiated by hyperparameters and timestamps in UI better than just random names. They also can be renamed manually in UI.

This PR proposes to remove the comparison `run_name` and `output_dir`, and **always** set the run name specified by the user.

Please, let me know what you think.

Env params:
```
- `transformers` version: 4.41.0.dev0
- Platform: Linux-5.15.0-1056-aws-x86_64-with-glibc2.31
- Python version: 3.10.9
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.2
- Accelerate version: 0.29.3
```

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30462). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I like the idea of doing `output_dir` as the `run_name`, but what if someone wants to stay with a random name? Probably it is better to stay with the same behavior as W&B has, if run_name is specified we set it, if not it will be generated randomly.

@parambharat we would much appreciate your thoughts about this change, please let us know if we are missing any of your use cases as you were one of the contributors for this callback 🤗 

cc @parambharat (right @)

@qubvel : This check was quite deliberate. The `TrainingArguments` by default sets the `run_name` descriptor to the `output_dir` if the user doesn't set a `run_name` see [here](https://github.com/huggingface/transformers/blob/d1d94d798f1ed5c0b5de9a794381aeb7dc319c12/src/transformers/training_args.py#L1557-L1558). 

That's why the condition [here](https://github.com/huggingface/transformers/blob/d1d94d798f1ed5c0b5de9a794381aeb7dc319c12/src/transformers/integrations/integration_utils.py#L766) checks if the user deliberately set the `run_name` i.e. it's not the same as the `output_dir` Otherwise, it defaults to the `output_dir` in which case the W&B run would be the `output_dir`. Therefore, if they, don't set a `run_name` we default to generating a random name in W&B.

@parambharat thanks a lot for the clarification, I didn't notice that `run_name = output_dir` assigned in training arguments!

There might be the following cases with the current code:
 
1) A user **dont** set `run_name` -> they have a random run name in W&B UI, **but** `run_name = output_dir` in training arguments
2) A user set `run_name = output_dir` -> they have **unexpected** random run name in W&B UI, and `run_name = output_dir` in training arguments
3) A user set `run_name != output_dir` -> they have the same name in W&B UI and in training arguments

I suppose we can resolve inconsistency in 1st and 2nd with one of the following ways:
1) Remove assigning `run_name = output_dir` in training arguments if user did not set `run_name` (its not working for W&B anyway), and remove the comparison `run_name == output_dir` in W&B logger. We will get predictable behaviour: if `run_name` is set we will get it in UI, otherwise, we will get a random name. But this will change behaviour for MLFlowCallabck, now its directly using `run_name` without comparison with `output_dir`.
2) Just remove the comparison `run_name == output_dir` in W&B callback. If `run_name` is set we will get it in UI, otherwise, we will get `output_dir` as a run name in W&B UI. No option to get a random name, but I would not expect that there would be too many runs with the same `output_dir`, so this case is also acceptable. And it will be consistent with MLFlowCallabck.

@muellerzr @parambharat @amyeroberts what do you think?

I like option 2. Option 1 I think is better overall (arguments' behaviour is more consistent), but we want to avoid an unexpected change in behaviour in MLFlowCallback. 

I loved my random run names =(

same, now they're lame `./output/path` every time. At least when W&B controlled the random run names, they'd index them automatically. Now it's impossible to tell without having to manually configure the run name each time.


Hi @winglian @tcapelle thanks for letting this know!
Sorry, that random names were removed, would you like to suggest a fix/PR for that?
You can probably use a hack to return random names with `training_args.run_name = None`