`TrainingArguments` `lr_scheduler_type="cosine_with_restarts"` can/does not pass a `num_cycles` argument to `get_cosine_with_hard_restarts_schedule_with_warmup()`
Author: hogru
Pull request: no

### System Info

- `transformers` version: 4.24.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.9.12
- Huggingface_hub version: 0.11.0
- PyTorch version (GPU?): 1.12.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I paste some dummy code but I think the explanation is more important (unless I have overlooked something): The `lr_scheduler_type="cosine_with_restarts"` that I pass to the `TrainingArguments` is used to call `get_scheduler()` in `optimization.py`. There it's mapped to `get_cosine_with_hard_restarts_schedule_with_warmup()`, but without a `num_cycles` argument, defaulting to `1`, i.e. it behaves like the `cosine` option.

Probably I could build the scheduler myself and pass it to the `Trainer`, but then I need to calculate the `num_trainings_steps` myself, correct? If true, would `len(train_dataset) * num_epochs // batch_size // gradient_accumulation_steps` be a decent approximation?

```python
args = TrainingArguments(
    output_dir="./checkpoints",
    per_device_train_batch_size=128,
    per_device_eval_batch_size=128,
    evaluation_strategy="steps",
    eval_steps=1_000,
    logging_steps=1_000,
    gradient_accumulation_steps=8,
    num_train_epochs=50,
    weight_decay=0.1,
    warmup_steps=5_000,
    lr_scheduler_type="cosine_with_restarts",  # that's actually the only relevant line
    learning_rate=5e-4,
    save_steps=1_000,
)

trainer = Trainer(
    model=model,
    tokenizer=tokenizer,
    args=args,
    data_collator=data_collator,
    train_dataset=tokenized_data["train"],
    eval_dataset=tokenized_data["validation"],
)

trainer.train()
```

### Expected behavior

Passing `lr_scheduler_type="cosine_with_restarts"` should allow for an additional parameter `num_cycles` in `TrainingArguments` which should then be passed on to `get_cosine_with_hard_restarts_schedule_with_warmup()`.

Comments:
Yes, there is no argument to pass that information, so in this instance you should either build the scheduler yourself and pass it, or subclass the `Trainer` to override the `create_scheduler` method, whichever you prefer.

In both cases the formula you passed should give the good number of training steps!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I believe this should be fixed at a certain point!

Is there any example to build scheduler using ```get_cosine_with_hard_restarts_schedule_with_warmup()``` and pass it to the Trainer by including it in the ```TrainingArguments```? 

I don't know when it was implemented, but you can now set it like this,
```
lr_scheduler_type="cosine_with_restarts",
lr_scheduler_kwargs={"num_cycles":3},
```