`trainer.evaluate` always creates a new MLFlow run, separate from the one used during `train()`
Author: nathan-az
Pull request: no
Labels: name: Feature request

name: bug

### System Info

`transformers` version: 4.46.1
`mlflow==2.18.0`

I have confirmed this on my local machine (mac) and our training cluster (8x H100 using standard nvcr image), using a local and external MLFlow tracking server, so this is not likely to be very environment dependent outside the above.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```py
from time import sleep

from transformers import Trainer, TrainingArguments, AutoModelForSequenceClassification, AutoTokenizer
from datasets import load_dataset
import mlflow
import os


with mlflow.start_run() as run:
    run_id = run.info.run_id
os.environ["MLFLOW_RUN_ID"] = run_id
print("Original run id:", run_id)

# Load a pre-trained model and tokenizer
model_name = "distilbert-base-uncased"
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Load a small dataset
dataset = load_dataset("imdb", split="train[:100]")

# Tokenize the dataset
def tokenize(batch):
    return tokenizer(batch['text'], padding=True, truncation=True, max_length=128)

tokenized_dataset = dataset.map(tokenize, batched=True)

# Define training arguments
training_args = TrainingArguments(
    output_dir="./results",
    evaluation_strategy="epoch",
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    num_train_epochs=1,
    logging_steps=1,
    report_to="mlflow"
)

# Initialize Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset,
    eval_dataset=tokenized_dataset,
)

# Train the model
trainer.train()

# Evaluate the model
eval_results = trainer.evaluate()
print(eval_results)

sleep(10)
runs_dir = "./mlruns/0"
runs = [x for x in os.listdir(runs_dir) if x != "meta.yaml"]
for run_id in runs:
    print(f"Metric files for {run_id}")
    metrics_path = os.path.join(runs_dir, run_id, "metrics")
    print(*os.listdir(metrics_path), sep="\n")
    print("="*16)
```
Running the above script should confirm that two separate runs are created, and that metrics files are placed in each.

### Expected behavior

I expected that all logging would be pushed to the existing run ID (unless there is a reason that this is not the case).

Apologies if this is documented behaviour, but I could not find a reason.

Comments:
cc @muellerzr @SunMarc 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@nathan-az do you have an idea of the fix? If so, feel free to open a PR with the fix, we're missing some bandwith in the new year. Thanks 🤗 

@LysandreJik I'm starting to think having a separate run is actually a good thing, for a couple of reasons:
* The trainer can store the same metrics in time series during `.train`, and we don't want to overwrite these during `.evaluate`
* I often use one tracking run for multiple stages of model training (to co-locate artifacts)

A middleground her might be for `evaluate` to still create a new run, but for it to be a child run if an existing `MLFLOW_RUN_ID` is set in the environment, at least improving organisation. I'm happy to take a look, but won't get around to it for a few weeks. 

Feel free to assign to me, I'll link this issue in the eventual draft PR 🤗 .

As Lysandre is off for a couple of weeks, I assigned you to the issue ! Excited to see the draft PR 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Apologies @SunMarc - I haven't found the time to get to this and am unlikely to any time soon. I'd encourage anybody interested to take a look into this!

As mentioned there might have to be a bit of discussion around the best course of action, given that the user likely doesn't want to overwrite the metrics written during training. Two options I can think of:
* Use the existing run but prefix the metric names to make them unique, or
* Create a child run if an existing run ID is found