Unified transformer interface
Author: ZhaofengWu
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

As we're called `transformers`,  it would be nice if there's a `Transformer` class that is not associated with any pretrained model and that we can directly instantiate, like
```python
from transformers import Transformer, Seq2SeqTransformer
transformer = Transformer(n_layers=8, dim=512)
seq2seq_transformer = Seq2SeqTransformer(n_enc_layers=6, n_dec_layers=8, dim=512)
```
They could follow a modular design that makes swapping components very easy, e.g.
```python
class SelfAttention:
    ...

class Transformer:
    def __init__(self, self_attention_cls=SelfAttention, n_layers=8, ...):
        ...
```

## Motivation

1. Makes it easier for researchers to modify the transformer architecture and train it from scratch.
2. Allows great refactoring of the current `modeling_xxx.py` files. Often these files are highly identical minus a few minor differences. For example, the embeddings, self-attention layers, etc. are often identical across multiple models, but the current design does not allow much code re-use. With this change, I imagine a lot of the current `modeling_xxx.py` files can be reduced significantly.
3. A side-effect of this refactoring would be increased readability. Without much repetitive code, it would be much clearer to see the differences between architectures, which would be helpful for many users.

Comments:
Hi @ZhaofengWu, thank you for proposing this feature. 

You're noting a decision that we've made very early on in the design of the `transformers` library, which is to have standalone model files, **without any abstraction**, including the `Transformer` abstraction you're noting here. We believe it's much easier to read and understand a model file if there is no abstraction. Understanding a model file is done by reading the model file. There are no other files to go to to understand it, simply a single model file. This is our objective, as is described in the [philosophy](https://huggingface.co/transformers/philosophy.html).

You mention having a `Transformer` class which is not associated with any specific architecture, but how would we do that? Would it be a Transformer that has absolute positional embeddings, or relative? Would it accept token type IDs? Would it have a pooler layer? How would it perform its attention? All of these questions have answers that mean we're defining a specific `Transformer` architecture, which is not something we want to do.

Finally, I believe we already have what you're looking for; if what you want to do is to have a transformer that you can tweak, so as to implement your own architecture, I recommend you take a look at the [model templates](https://github.com/huggingface/transformers/tree/master/templates/adding_a_new_model), which add a "bland" model which you can customize to your needs. It also adds the tests (which pass), according to the naming you defined, and places the import where they should be, alongside the auto classes.

Thank you for your response! I was thinking about the `Transformer` abstraction in pytorch, fairseq, etc. Re. not associating with any specific architecture, one solution could be to implement all options in `Transformer` and allow clients to control which one to use with flags. Of course, there's no such thing as "all options" as researchers develop new architectural improvements, and we would need to be constantly adding to this class, which would not be ideal. Nevertheless, for some common boolean stuff (e.g. token type IDs), perhaps it's not the worst idea to implement it there and have a flag to control if it is included. For most other stuff, the modularity I mentioned could help. If we can swap components of `Transformer` easily, we can only write small pieces of code while reusing the rest of the architecture. We can keep some "default" choices in there which other models can override (e.g. transformer-xl has a relative positional embedding class (this implementation can also be shared across models) that overrides the absolute one in `Transformer`), but if we are committed to making the class absolutely not associated with _any_ architecture, it can be made an abstract class.

Re. model templates, they are nice, but in my understanding, it's similar to copying the entire architecture from some other model file, e.g. BERT, and modify it. This would still result in a lot of boilerplate code, even if the researcher/developer doesn't have to spend time writing it. The software engineer in me often screams when I have to copy massive amounts of architectural code which causes huge redundancy. And when I am reading the implementation of a new model, I also often don't like this boilerplate as it makes the real differences hard to notice. Of course, reading the paper is one thing, but there are often small engineering details not mentioned in the paper. This redundancy was the main motivation for this issue. Sometimes I have to do python hacks (thankfully we're not doing this in C) to accomplish code re-use.

But, of course, I definitely respect your philosophy. I'm sure there must be challenges that I'm not seeing, and this change would definitely be huge. If there are no ideal ways to do this at the moment, please feel free to close this issue.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.