GPT2DoubleHeadsModel made parallelizable
Author: ishalyminov
Pull request: yes

# What does this PR do?

GPT2DoubleHeadsModel made parallelizable; it is also reflected in the test_modeling_gpt2.py suite

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
@patrickvonplaten , @LysandreJik

Also pinging @stas00 here - does it make sense to add parallelize ability to GPT2DoubleHeadsModel?

If it's being used then yes since `GPT2LMHeadModel` has it.

@alexorona, do you want to take a look at this?

@stas00 yeah, so I've been using the `GPT2DoubleHeadsModel` for my tasks since reading [this medium](https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313) (I guess a lot of people in the dialogue community also followed that tutorial).
And seeing `parallelize()` implemented with the `GPT2LMHeadModel` got me curious to have the double-headed one working like that as well :)

@LysandreJik, we parked any further activity on extending naive MP to other `transformers` models and their sub-classes because while it solved the immediate need it's not a good long term solution due to very inefficient gpu utilization.

We are working on integrating ZeRO-3 from DeepSpeed and fairscale which will automatically solve this scalability issue and will make the naive MP approach redundant and we can then decide whether it makes sense to keep it.

Until we sort it out and we can reliable know that ZeRO solves this problem and it's accessible to all users you can definitely merge this since @ishalyminov clearly has a good use for it.

@ishalyminov Great work here! You're bringing up an important point. I've had this some question myself. @thomwolf It might be useful to add a few sentences to [this Medium article](https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313) clarifying whether distractors are likely to improve final performance in tasks that are ultimately concerned with generative text. It's not entirely clear if your approach using distractors and a double-headed model is an artifact of the competition setup or whether it's an approach you would recommend for anyone trying to fine-tune a transformer for chatbot-style tasks. If someone only cares about language modeling, do you think a double-headed approach with distractors and a classification task would usually produce a better chatbot than simply focusing on LM? Does it matter if the chatbot is attempting to model several discrete personalities present in the dataset?

@alexorona thanks! Yeah it would be very interesting to hear more about @thomwolf's ConvAI experience:) As for me, I didn't conduct any evaluation of how the NSP task affects the resulting LM (and indeed, there are some works out there that don't use this secondary task at all).

But for what it's worth, we found the NSP head to be beneficial for hybrid [generative/retrieval](https://github.com/microsoft/GRTr) GPT-2 based dialogue architectures.
Also I guess the multi-task setup makes the model intuitively more versatile for a range of downstream tasks, as was originally proposed in the [BERT paper](https://arxiv.org/pdf/1810.04805.pdf) - would be really useful if there was an experimental evaluation proving or disproving this for the case of ConvAI GPT-2.