Jia multi gpu eval
Author: liyongsea
Pull request: yes

# What does this PR do?
This PR provdie a script to run codeparrot human eval in multi GPU mode
The code is a enhanced version of human_eval.py

# Performance data
The performance is check using the following command
```
accelerate launch scripts/human_eval.py --model_ckpt lvwerra/codeparrot-small --do_sample True --temperature 0.2 --top_p 0.95 --n_samples=40 --HF_ALLOW_CODE_EVAL="1" --device_int=0 --num_tasks=8 --batch_size=10
```
One should obtain something similar to 
```
Results: {'pass@1': 0.1375, 'pass@10': 0.21251641317430792}
```

# Time benchmark
First configure accelerate
```
accelerate config
```

Remember to put more than 1 process if want to leverage multip gpu.
Then run
```
accelerate launch scripts/human_eval.py --model_ckpt lvwerra/codeparrot-small --do_sample True --temperature 0.2 --top_p 0.95 --n_samples=200 --HF_ALLOW_CODE_EVAL="1" --batch_size=10
```
One should obtain something similar to 
```
Results: {'pass@1': 0.03807926829268292, 'pass@10': 0.056158491636524525, 'pass@100': 0.0739473539338153}                                                                                                  
```

Here is time performance benchmark under a GPU VM with 4xT4
| Number of processes | Evaluation time for lvwerra/codeparrot-small | 
|---------------------|----------------------------------------------|
| 1                   | 2:10:00                                      |  
| 4                   | 35:00                                        | 

You could also test the large model by setting `--model_ckpt lvwerra/codeparrot`, you might probably need to set a small batch size (1 is usually adapted)
Warning: when testing with the large model, accelerate is loading the model on all the processes. You would need about 60G RAM on your VM. (I am not sure there is another alternative)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hey @liyongsea, thanks for your PR! Could you rebase/merge on `main` so that the CI passes? Thank you!

There seems to be also a minor issue with the code style. The CI enforces some standards and you can fix any formatting issues with:

```bash
make fixup
```

For more information on both rebasing/merging and quality checks you can checkout the [contributing guide](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

> There seems to be also a minor issue with the code style. The CI enforces some standards and you can fix any formatting issues with:
> 
> ```shell
> make fixup
> ```
> 
> For more information on both rebasing/merging and quality checks you can checkout the [contributing guide](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

Thank you for your feedback @lvwerra here is what I am going the do:
- [x] Rebase and fix the formatting issue
- [x] use task id to reorganize the output
- [x] remove padding before generation to avoid the 'n_copies' tricks

hi @lvwerra 
I finished the implementation. Please start review. Here are some area you might need to pay extra attention.
- I keep the behavior of sending copies of prompts, such that when generating 1M prediction per prompt, the same task is also distributed across worker
- Padding is removed before generation. However, the code still requires n_tasks x n_samples to be multiple of n_processes. Because the length of the dataloader should be a multiple of n_processes (do you know how to easily avoid this). Do you think it is ok? Or shall I change the behavior?

@lvwerra Thank you for your suggestions. I learnt a lot ! It is integrated into the PR, I will test it.

> Regarding your second point: I don't think this is true - accelerate should be able to handle that! It just means that during the last batch of jobs not all GPUs will be utilized but I think that is fine. Did you run into any problems? If not we could remove the ValueError.

I have an error message when doing so, I will dig into it

Hi @lvwerra I updated the code according to your suggestion. And I remove the exception regarding the division issue (please see my comment above).
I really appreciated your time. Tell me if you spot anything.

You can go the the branch [empty_tensor](https://github.com/liyongsea/transformers/tree/empty_tensor), it is the same PR as this one by remove the if condition and an exception.
Then config accelerate to 4 process and run
```
accelerate launch scripts/human_eval.py --model_ckpt lvwerra/codeparrot-small --do_sample True --temperature 0.2 --top_p 0.95 --n_samples=10 --HF_ALLOW_CODE_EVAL="1" --num_tasks=6 --batch_size=10
```
You will notice something like
```
TypeError: : only integer tensors of a single element can be converted to an index
```
Where accelerate is sending an empty tensor to some processes during the last iteration

I will also take a look into accelerate to see if I find anything

@lvwerra for the empty tensor issue, I found this in the documentation
https://github.com/huggingface/accelerate/blob/1d95ebdaa40ed42d0bb2b41ba5531b49eb8948b1/src/accelerate/data_loader.py#L204
I am really not sure if it is related. Tomorrow I will make a smaller example to verify this

Hi @liyongsea , thank you for your contributions. I made a test run on my side and the code works fine!
For an NVIDIA Tesla A100 with 2 GPUs and the parameters you're using, I get the following results
|#processes | Time | pass@1 | pass@10| pass@100| 
|   -   |   -  |   -  |    -  |    -  | 
| 1 | 3:07:26 | 3.41% | 5.39% | 7.01% |
| 2 | 1:36:34 | 3.51% | 5.63% | 6.67% |

So as the numbers seem to add up I think we can merge this once the `set_seed` is added. The check for empty batches is simple enough so we can leave it like that for now in my opinion - no reason to spend much time on that. Thanks again for adding this! 

> So as the numbers seem to add up I think we can merge this once the `set_seed` is added. The check for empty batches is simple enough so we can leave it like that for now in my opinion - no reason to spend much time on that. Thanks again for adding this!

Hi @lvwerra 
I have finished the modifications.
- accelerate.utils.set_seed is used with device_specific=True
- I have updated the accelerated version which solve the empty tensor issue

Feel free to run the whole script again and merge if you are ok !