Documentation of Adafactor is at odds with Google implementations
Author: martiansideofthemoon
Pull request: no

### System Info

- `transformers` version: 4.22.0
- Platform: Linux-5.4.0-126-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.9.1
- PyTorch version (GPU?): 1.12.0+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes, two RTX8000
- Using distributed or parallel set-up in script?: Yes, DDP via HuggingFace accelerate

### Who can help?

documentation of Adafactor: @sgugger @Narsil 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The documentation of Adafactor seems to be at odds with the Google implementation in T5X / PaLM. I've found these hyperparameters to be critical while optimizing HuggingFace transformers for metric learning tasks. Specifically the documentation ([link](https://huggingface.co/docs/transformers/main_classes/optimizer_schedules#transformers.Adafactor)) says `Use scale_parameter=False` and `Additional optimizer operations like gradient clipping should not be used alongside Adafactor`.

However, in T5X the default hyperparameter is set to `True` and is not modified in the config files (https://github.com/google-research/t5x/blob/83046e22750635f76c7e600f01c0a002915b52b8/t5x/adafactor.py#L199).

Similarly, PaLM used `scale_parameter` with a constant learning rate, 

```
Optimizer – ... This is effectively equivalent to Adam (Kingma & Ba, 2014) with “parameter scaling,”
which scales the learning rate by the root-mean-square of the parameter matrix. Because the weight
initialization is proportional to 1/√n, the effect of this is similar to the manual scaling down of Adam
learning rate as in Brown et al. (2020). However, parameter scaling has the benefit that parameter
matrices which operate at different scales (the embeddings and layer norm scales) do not have their
learning rate scaled down at the same rate.... We use an Adafactor learning rate of 10−2 for the first 10,000
steps, which is then decayed at a rate of 1/√k, where k is the step number. We train with momentum
of β1 = 0.9 .... We use global norm gradient clipping (Pascanu et al. (2012)) with a value of 1.0 for all
models...
```

Overall, consistent with the Google recommendations, the following hyperparameters worked well for me:

```
optimizer = Adafactor(model.parameters(), scale_parameter=True, relative_step=False, warmup_init=False, lr=float(args.learning_rate))
...
accelerator.clip_grad_norm_(model.parameters(), 1.0)
```



### Expected behavior

N/A (documentation fix)

Comments:
Sorry I'll skip this, I'm not very well versed in that area.

What led you to ping me if it's not too much to ask ? (Since maybe there are better people to ping here).

hi @Narsil thanks and no worries! I didn't find a section on optimizers in the issue builder, so I pinged the people in the closest two areas (trainer and pipeline). I am guessing "trainer" / @sgugger may be better able to answer the issue.

Transformers is not a library of optimizers, so you should really use an implementation of `Adafactor` form somewhere else that suites your need. It will be deprecated and removed in future versions :-) (Note that it comes from fairseq originally, so that's probably the reason you have comments at odds with T5x)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.