Issue 1:
Transformers has a Do Repeat Yourself policy in the sense that it does not provide building blocks that we then mix and match, but we strive to have each model be self-contained in terms of code, at the price of code duplication. You can find more about this philosophy in this blog post.

There are instances in the library (mostly with older models) where this is not respected. This issue will serve as a tracker for all those instances, so that the library is cleaner and each model/tokenizer/config is easier to tweak by itself. This will also make it easier for us to test individual models in autonomy.

If you wish to make a contribution to Transformers, you can help! Pick a config/model/tokenizer in the list below (double-check someone is not working on it already by searching this page!) and indicate with a comment that wish to work on it. Read our contributing guide as well as the section below, and once you are ready, open a PR and tag @sgugger on it.

How to remove a dependency from another model

There are two different types of dependencies: either a configuration/model/tokenizer uses an intermediate object from another model (example: some tokenizer uses the BasicTokenizer defined in the tokenization_bert module, or it subclasses another configuration/model/tokenizer.

In the first case, the object code should just be copied inside the file, with a "Copied from" statement. This will make sure that code is always kept up to date even if the basic object is modified. For instance, if a tokenizer is using BasicTokenizer, go copy the code in tokenization_bert for that class, then paste it in the tokenizer module you are treating and add the following copied from comment:

# Copied from transformers.models.bert.tokenization_bert.BasicTokenizer
class BasicTokenizer(object):
...

In the second case, the code of the class (and all its building blocks) should be copied and renamed to be prefixed by the model: for instance if you are copying code from the modeling_bert module to build Roberta, you replace all BertLayer, BertOutput etc... by RobertaLayer, RobertaOutput...

You should then add a copied from statement (when the copy is without any modification) like this one:

# Copied from transformers.models.bert.modeling_bert.BertAttention with Bert->Roberta
class RobertaAttention(nn.Module):
...

Note the replacement pattern that will adapt all names used. Note that:

you can add more of those patterns, separated by a comma like here.

you can ask to replace all possible casings like here

you can just copy one method and not the whole class like here

NB: No need for copied from statements in the config (the defaults are probably different anyway).

Objects to cover

Configurations

Flaubert config (should not use XLM)

LayoutLM config (should not use Bert)

LongformerConfig (should not use Roberta)

MarkupLMConfig (should not Roberta)

RobertaConfig (should not use Bert)

XLM-ProphetNet config (should not use ProphetNet)

XLM-Roberta config (should not use Roberta)

Models

BertGeneration (should not use BertEncoder)

Camembert (should not use Roberta) (PyTorch + TF)

Flaubert (should not use XLM) (PyTorch + TF)

mT5: PyTorch, TensorFlow, Flax (should not use T5)

XLM-ProphetNet (should not use ProphetNet)

Xlm-Roberta: PyTorch, TensorFlow, Flax (should not use Roberta)

Tokenizers

BertJapanese (should not use any imports from tokenization bert)

Blenderbot (should not use Roberta) (slow/fast)

Clip (should not use BasicTokenizer from Bert)

ConvBERT (should not use Bert) (slow/fast)

Cpm tokenizer (should not use XLNet) (slow/fast)

Derberta tokenizer (should not use GPT2) (slow/fast)

DistilBert (should not use Bert) (slow/fast)

Electra (should not use Bert) (fast)

Flaubert (should not use XLM)

Funnel (should not use Bert) (slow/fast)

Herbert (should not BasicTokenizer from Bert and XLM)

LayoutLM (should not use Bert) (slow/fast)

LED (should not use BART) (slow/fast)

Longformer (should not use Roberta) (fast tokenizer)

Luke (should not use Roberta)

Lxmert (should not use Bert) (slow/fast)

MobileBert (should not use Bert) (slow/fast)

Openai-GPT (should not use BasicTokenizer from Bert)

ProphetNet (should not use BasicTokenzier and WordPieceTokenizer from Bert)

Retribert tokenizer (should not use Bert) (slow/fast)

Roformer tokenizer (should not use any imports from tokenization bert)

Squeezebert tokenizer (should not use Bert) (slow/fast)

Issue 2:
🚀 Feature request

Over at EleutherAI we've recently released a 20 billion parameter autoregressive gpt model (see gpt-neox for a link to the weights). It would be great to get this into transformers!

Motivation

gpt-neox library is not quite as user-friendly as transformers, and is designed for efficient large scale training over inference. So we think integrating it into transformers would be great for accessibility.

Your contribution

Personally, I can dedicate a bit of time into integrating this model. We already have a PR to convert the weights to HF format, although:

I suspect you would want to introduce a new model class for it.

It is not merged / thoroughly tested with all possible configurations yet. GPT-NeoX has a bunch of configuration options, and it might be more straightforward to focus on just introducing a model class for GPT-NeoX-20B (which should largely be similar to GPT-J, with some caveats, see next section)

Difficulties

Whilst we do have a script to merge model parallel checkpoints that will be merged soon, in larger models we see some performance loss when merging mp ranks, due to there being some very slight differences in the replicated parameters between ranks (see the thread above for more details). If we want to integrate GPT-NeoX-20B as a model to be used on a single GPU, we need to figure out how to address this. I'm not sure if this bug is specific to neox, or was introduced in megatron or deepspeed, but I believe the bigscience team are also looking at merging mp models, so I guess we'll find out soon.

If we do integrate the model without model parallelism - it will be too large to run on most consumer GPUs. During inference, it takes ~45GB of GPU memory to run, and during training much more.