[Feature request] Introduce GenericTransformer to ease deployment of custom models to the Hub
Author: jordiae
Pull request: no

# 🚀 Feature request

Introduce a GenericTransformer model that can handle many different variants and tweaks of the Transformer architecture. There are 2 ways of doing this and I'm not 100% sure of which one would better suit HF:

1. Introduce a GenericTransformerModel with many different options (extensive config file), such as different positional embeddings or attention variants. The modeling code would constantly be updated by HF or contributions from the community and would be included in each release of the library itself. Backward compatibility would not necessarily be an issue if all new additions were disabled by default in the config class. Also, the model could be designed in a modular way to ease the addition of new variants (see torchtext's MHA container https://github.com/pytorch/text/blob/main/torchtext/nn/modules/multiheadattention.py).
2. Allow users to submit code following HF's interfaces alongside checkpoints. GenericTransformerModel would dynamically download and load code from the hub.

I think the first one would be more convenient to avoid third-party dependencies and potentially unsafe code. The second one would be way more flexible, though.

## Motivation

An important point of the HF Transformers library philosophy is outlined in the README of the repo:

> Why shouldn't I use transformers?
> This library is not a modular toolbox of building blocks for neural nets. The code in the model files is not refactored with additional abstractions on purpose, so that researchers can quickly iterate on each of the models without diving into additional abstractions/files.

To clarify, this feature request does NOT intend to modify this philosophy. which clearly has many advantages. Instead, it has the purpose of potentially alleviating one of the drawbacks of this philosophy: the difficulties in sharing custom models, even if these models just introduce small tweaks (see https://github.com/stanford-crfm/mistral/issues/85, https://github.com/huggingface/transformers/pull/12243).

This would hopefully encourage researching different variants and combinations. In case one variant stabilized as a well-defined architecture that was worth using, then it might be considered to add it to the library the "classical" way, having a specific class, documentation, etc.

## Your contribution

I can't allocate time to this at the moment. Sorry about that.


Comments:
cc @sgugger regarding 2. :)

The plan is to add in the coming weeks support for custom models directly in the AutoModel classes, with the user providing the code of their models in a modeling file in the same repository on the model hub (same for custom tokenizers).

ETA for this feature should be end of next week.

> The plan is to add in the coming weeks support for custom models directly in the AutoModel classes, with the user providing the code of their models in a modeling file in the same repository on the model hub (same for custom tokenizers).
> 
> ETA for this feature should be end of next week.

Perfect, thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

PR https://github.com/huggingface/transformers/pull/13467 introduced a first version of what you were asking for @jordiae! Let us know if it works for you :) 

> PR #13467 introduced a first version of what you were asking for @jordiae! Let us know if it works for you :)

Cool! Thanks!