[trainer metrics] fix cpu mem metrics; reformat runtime metric
Author: stas00
Pull request: yes

This PR improves and fixes trainer metrics:

* reworks the general ram memory tracking replacing `tracemalloc`, with "sampling" via `psutil` - in particular for peak tracking using a thread. `tracemalloc` proved to not track anything but python memory allocations, so we were missing most of the general RAM from reports. Now we are reporting much more. (other than swapped out memory).
* adds important details to memory metrics docs
* moves `psutil` dependency from just-for-tests to the core. I tried to find a built-in python equivalent, but the closest that I found is `resource.getrusage(resource.RUSAGE_SELF).ru_maxrss` which doesn't report what we need and it's not cross-platform.
* reformats secs to be in `hh:mm:ss.msec` format so it's much easier to read the runtime metric

Discovered the `tracemalloc` limitation while tracking a huge memory leak in DeepSpeed when re-using deepspeed in the same process. My tests were consuming hundreds of MBs of general RAM and the metrics were reporting nothing.

before:
```
BS=4; PYTHONPATH=src USE_TF=0 python examples/seq2seq/run_translation.py --model_name_or_path \
t5-small --output_dir /tmp/zero3 --overwrite_output_dir --max_train_samples 64 --max_val_samples 64 \
--max_source_length 128 --max_target_length 128 --val_max_target_length 128 --do_train \
--num_train_epochs 1 --per_device_train_batch_size $BS --per_device_eval_batch_size $BS \
--learning_rate 3e-3 --warmup_steps 500 --predict_with_generate --logging_steps 0 --save_steps 0 \
--eval_steps 0 --group_by_length --adafactor --dataset_name wmt16 --dataset_config ro-en \
--source_lang en --target_lang ro --source_prefix "translate English to Romanian: "

***** train metrics *****
  epoch                      =    1.0
  init_mem_cpu_alloc_delta   =    3MB
  init_mem_cpu_peaked_delta  =    0MB
  init_mem_gpu_alloc_delta   =  230MB
  init_mem_gpu_peaked_delta  =    0MB
  train_mem_cpu_alloc_delta  =   60MB
  train_mem_cpu_peaked_delta =    0MB
  train_mem_gpu_alloc_delta  =  232MB
  train_mem_gpu_peaked_delta =  472MB
  train_runtime              = 5.5261
  train_samples              =     64
  train_samples_per_second   =  1.448
```

after this PR:

```
***** train metrics *****
  epoch                      =        1.0
  init_mem_cpu_alloc_delta   =     1298MB
  init_mem_cpu_peaked_delta  =      154MB
  init_mem_gpu_alloc_delta   =      230MB
  init_mem_gpu_peaked_delta  =        0MB
  train_mem_cpu_alloc_delta  =     3446MB
  train_mem_cpu_peaked_delta =        0MB
  train_mem_gpu_alloc_delta  =      232MB
  train_mem_gpu_peaked_delta =      472MB
  train_runtime              = 0:00:05.66
  train_samples              =         64
  train_samples_per_second   =      1.412
```

@sgugger, @LysandreJik 

Comments:
Woops, didn't mean to approve

> Thanks for the fix! Like @LysandreJik I would avoid adding `psutils` as a main dependency. We can have it come with a `is_psutils_available` and only compute the mems metrics when it's there.

Oh, I was thinking to assert to say to install it. They can disable the mem metrics flags if they don't want to install it.

So we have:

A. `assert("pip install psutil to use memory metrics")`
B. `return if not is_psutils_available()`


Either way works for me.


I would use option B personally. Having an error because something is not installed is not something we like (cf wandb).

looks like I did something wrong with the runtime metrics - checking.
``` 
Trainer is attempting to log a value of "0:00:02.18" of type <class 'str'> for key "train/train_runtime" as a scalar. This invocation of Tensorboard's writer.add_scalar() is incorrect so we dropped this attribute.
```

**edit**: fixed

@sgugger, so the only missing part is how can the user reach the extensive docs in `TrainerMemoryTracker` docstring? It's sort of an internal class. 

Perhaps I should move the docs elsewhere so that the user can understand what the memory metrics are? Perhaps I can do the following:

1. move the bulk of  the current `TrainerMemoryTracker`  docstring explaning the metrics to `save_metrics` and then it'll automatically be documented in the right place
2. and add a note to `log_metrics` docstring to read the docstring of `save_metrics` for details
3. and add a note to `TrainerMemoryTracker` docstring to read the details in `save_metrics` for details

I think your approach for making the doc more visible is a good one, so I'm fine with it. Also add there that `pip install psutil` is necessary to get the memory metrics?

> I think your approach for making the doc more visible is a good one, so I'm fine with it. 

Great!

> Also add there that `pip install psutil` is necessary to get the memory metrics?

It's already there ;)

Ah, missed it. Sorry about that!

OK, docs moved/reshaped/cross linked from 2 places. Decided to put the main doc in `log_metrics` since that's where they are most "visual". 

If you could get one last look at the final version of the docs, that would be great. I expanded it a little bit more. I checked that they render well and cross-reference is a working link.

Hmm, I'm having second thoughts about skipping and not asserting if `psutil` is unavailable. Since there is a function flag to skip memory metrics, if the flag is `False` and we skip the metrics, that's not super intuitive. So if a user doesn't want the memory metrics they don't have to install `pustil` but can simply disable the metrics by setting the skip flag to `True`.

Perhaps it'd be agreeable with you to change the behavior to option A. in https://github.com/huggingface/transformers/pull/10937#issuecomment-809529412, i.e. to assert.

I'd personally really like to avoid the script failing even if you can set an argument to avoid that.