Experimental symbolic tracing feature with torch.fx for BERT, ELECTRA and T5
Author: michaelbenayoun
Pull request: yes

# What does this PR do?


This PR provides a function called "symbolic_trace" which enables symbolic tracing for models of the library using the new and still experimental torch.fx feature. Our models can't be symbolically traces directly using `torch.fx`, so this is wrapper function that overcomes various issues.
 
This new feature allows to perform [many kinds of transformations to the graph](https://pytorch.org/docs/stable/fx.html).

It's also needed for projects like https://github.com/flexflow/FlexFlow/

As an experiment currently only three models are supported: BERT, ELECTRA and T5 (support for other models will follow soon).

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Let's also:
1. add some basic usage doc - we can start with just docstring
2. add one test for one of the models, polish it and then see how to replicate it for other models.

My understanding was that this is experimental and as we start using this side of library we will generalize and improve things. Hence the more slack approach.

Same for tests, I thought it was good to start with unique tests because the workarounds are unique and then over time as more models are ported to come up with common tests. 

@michaelbenayoun, one way to approach this puzzle is to create common tests for what's the same in all of them, and if something is unique to a given model then have just that tested in that model's test file. If you need help with that, please don't hesitate to ask.


Even for experimental features like model parallelism, we are using common tests. This should not be different IMO.

@sgugger, Michael merged the custom tests into common_tests and significantly simplified the mods to the models - yay!

So it looks ready for your review whenever you have a chance. 

Thank you!

Sorry for jumping in. 
Out of curiosity, what is the scenario to use this symbolic tracing feature? Didn't find any example/doc...
Thanks.

Well, I initially wanted this in order to be able to try https://github.com/flexflow/FlexFlow, which requires symbolic tracing - but I haven't had a chance to do so yet.

Got it, thanks for the explanation.

> Sorry for jumping in.
> Out of curiosity, what is the scenario to use this symbolic tracing feature? Didn't find any example/doc...
> Thanks.

This would be also be helpful to quantize models using [ FX Graph Mode Quantization](https://pytorch.org/docs/stable/quantization.html?highlight=quantization) which automate the quantization process in Pytorch. 

Are these updates still functional currently? As no modeling_fx_utils.py can be seen in the source code directory.