ClearMLCallback enhancements: support multiple runs and handle logging better
Author: eugen-ajechiloae-clearml
Pull request: yes

Currently, training multiple models in the same script might cause some model checkpoints logged to ClearML to be lost, as well as scalars, when using ClearMLCallback. This PR fixes these issues. Scalar visualization in ClearML has also been enhanced. 

What the PR does:
1. We count the number of times `ClearMLCallback.setup` is called via class variables. When a second training run is created, we do one of the following: we create a new task that will be used to log all the models, metrics etc. OR we keep the same task and we suffix the metrics/checkpoint etc. with the setup number.
We keep the same task if the task was created externally or if we are running remotely. We create new tasks if `ClearMLCallback` is the one that created the first task (in this case we also close the task so we can create another one).
2. We now delete model checkpoints if `save_total_limit` is set and the limit has been exceeded.
3. We switched the title/series of the logged scalars for better visualization.
4. We now, by default, don't fetch the configurations/hparams from the backend when running remotely, as these can contain temp files or other variables that are related to the local environment. The user can still override these tho, by setting, in the UI/via scripts, `_ignore_hparams_ui_overrides_` or `_ignore_model_config_ui_overrides_` to False. 

Comments:
Hi @LysandreJik! Can you or someone from your team please review this PR?

Hi @eugen-ajechiloae-clearml, apologies for the delay. Reviewing now!

@eugen-ajechiloae-clearml We had some recent issues with new package releases and compatibility on our circle CI pipeline. Fixes have been merged into `main` rebasing and pushing the changes should resolve the tests that are currently failing 🤗 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28559). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts I believe I synced the branch appropriately and `tests_exotic_models` still fails. Is there something I'm missing?

@eugen-ajechiloae-clearml I'm not sure what's happening to be honest. From the commit history, it all looks good. And I'm pretty sure these failures are independent of the changes in this PR. I'm going to see if this is popping up elsewhere. 

OK, seems like there's some outstanding issues with package installs. There's a PR open to fix: #28834. Once that's merged in, a final rebase should do it :) 

This new implementation ClearML callback's output all metrics from log as multiple series at one plot, that's looks not readable. 
![image](https://github.com/huggingface/transformers/assets/6096108/875ccebc-fb97-45ac-b8cb-f70110865c32)
For example: runtime and samples_per_seconds has different scale than other metrics. 

Previous version report metrics as one metric - one plot.

Any ideas how can  we set / filter metrics without code changes?
As temporarily solution I add my own callback for send some metrics to ClearML at other plots.