Add `TimedTextStreamer` for measuring per-token latency using `generate()` 
Author: danielkorat
Pull request: yes

# What does this PR do?

Adds a `TimedTextStreamer` for measuring per-token latency using `generate()` .
The idea behind this implementation is a non-intrusive method to measure per-token generation latency, which does not modify code in `genration/utils.py` making it compatible with future versions of `utils.py` as well. 
By measuring per-token latency, one can deduce the 1st token latency vs the 2nd token latency (all tokens after the 1st), an important distinction in benchmarking of decoder only models.

## Usage

```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer, TimedTextStreamer

>>> tok = AutoTokenizer.from_pretrained("gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("gpt2")
>>> inputs = tok(["An increasing sequence: one, two, three, four, five, six, seven,"], return_tensors="pt")

>>> # Option 1: Do not print the generated text (more accurate measurement)
>>> streamer = TimedTextStreamer()
>>> _ = model.generate(**inputs, streamer=streamer, max_new_tokens=5)
>>> streamer.get_token_times()
[146.61671698559076, 91.94262302480638, 16.620049951598048, 14.585152035579085, 14.050466008484364]

>>> # Option 2: Print the generated text as well
>>> streamer = TimedTextStreamer(tokenizer=tok)
>>> _ = model.generate(**inputs, streamer=streamer, max_new_tokens=5)
 eight, nine, ten
>>> streamer.get_token_times()
[162.81271493062377, 18.371607991866767, 15.906393993645906, 14.754525036551058, 14.49775299988687]
```

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


cc @gante @younesbelkada 


Comments:
Hi @danielkorat 👋 Thank you for opening the PR!

I haven't seen other requests to measure per-token time, so I'm assuming a limited number of users would benefit from it. Moreover, I have in my list of TODOs for `generate()` to refactor how streaming works: instead of printing/emitting tokens, `generate()` would become an iterator -- from which per-token measurements would become trivial.

For the reasons listed above, I'm deciding not to merge this PR (limited benefits, short/medium time of existence, plus maintenance burden).

If other users feel strongly about this PR, please react to this comment: happy to reconsider when we reach 10 reactions 🤗  

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.