[CB] Add a "max_requests_per_batch" parameter
Author: remi-or
Pull request: yes

# Summary

This PR adds a `max_requests_per_batch` to the continuous batching config, which can be manually set by the user or auto-inferred from the number of requests passed.
Introducing this parameters create a clear difference between:
- the tensors that should be sized by the number of tokens in a batch, which we usually want to maximize during prefill
- the tensors that should be sized by the number of request in a batch, which usually is much lower

For instance, the logits tensor that the model outputs has size `[max_batch_tokens, vocab_size]`, which can be quite big because `vocab_size` is. We then perform operations on that tensor, notably converting it to `fp32` to match `generate`, which double its memory footprint and can cause CB to OOM.
To reduce its size, we can extract from it only the useful next token predictions, which we know there are only `max_requests_per_batch` of. This dramatically reduces the downstream memory footprint. 

Note: it would even be better to slice the tensor before the lm_head layer, but I am not sure this is possible. will look it up. 

## Performance

| label                |   main acc |   with PR acc |   main (tok/s) |   with PR (tok/s) | diff   |
|----------------------|------------|---------------|----------------|-------------------|--------|
| gsm8k_default        |      0.822 |         0.822 |        2431.32 |           2448.09 | +0.7%  |
| gsm8k_sampling       |      0.789 |         0.796 |        1938.81 |           1947.00 | +0.4%  |
| gsm8k_compile        |      0.822 |         0.822 |        2444.41 |           2454.80 | +0.4%  |
| gsm8k_no_fast_decode |      0.822 |         0.822 |        2353.78 |           2349.59 | -0.2%  |
| gsm8k_bare_bones     |      0.821 |         0.821 |        1814.19 |           1862.95 | +2.7%  |
| ifeval_default       |      0.447 |         0.447 |        7576.80 |           8293.33 | +9.5%  |
| few_blocks           |          - |             - |         699.38 |            691.24 | -1.2%  |
| multi_return_seq     |          - |             - |        1529.53 |           1630.70 | +6.6%  |
| rollouts_1024        |          - |             - |        3278.89 |           3285.90 | +0.2%  |
| rollouts_2048        |          - |             - |        3125.88 |           3130.02 | +0.1%  |
| rollouts_4096        |          - |             - |        2842.50 |           2789.79 | -1.9%  |
| rollouts_8192        |          - |             - |        2258.64 |           2259.52 | +0.0%  |
| rollouts_16384       |          - |             - |        1406.75 |           1397.03 | -0.7%  |

Overall some noise, with slight improvements for workloads that benefit from the added parameter (prefill heavy workflow) and a slight performance loos on RL 4096 (ran it again, went from 2845 tok/s to 3090 tok/s with a compile reset, so I am classifying this as compile cache poor performance). 

### Comment

Tests pass. Added one test for an edge case of the new feature.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46434). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46434)