Make all models folder independent from each other
Author: sgugger
Pull request: no
Labels: name: Good First Issue

name: HACKTOBERFEST-ACCEPTED

Transformers has a Do Repeat Yourself policy in the sense that it does not provide building blocks that we then mix and match, but we strive to have each model be self-contained in terms of code, at the price of code duplication. You can find more about this philosophy in [this blog post](https://huggingface.co/blog/transformers-design-philosophy).

There are instances in the library (mostly with older models) where this is not respected. This issue will serve as a tracker for all those instances, so that the library is cleaner and each model/tokenizer/config is easier to tweak by itself. This will also make it easier for us to test individual models in autonomy.

If you wish to make a contribution to Transformers, you can help! Pick a config/model/tokenizer in the list below (double-check someone is not working on it already by searching this page!) and indicate with a comment that wish to work on it. Read our [contributing guide](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md)  as well as the section below, and once you are ready, open a PR and tag @sgugger on it.

## How to remove a dependency from another model

There are two different types of dependencies: either a configuration/model/tokenizer uses an intermediate object from another model (example: some tokenizer uses the `BasicTokenizer` defined in the `tokenization_bert` module, or it subclasses another configuration/model/tokenizer.

In the first case, the object code should just be copied inside the file, with a "Copied from" statement. This will make sure that code is always kept up to date even if the basic object is modified. For instance, if a tokenizer is using `BasicTokenizer`, go copy the code in `tokenization_bert` for that class, then paste it in the tokenizer module you are treating and add the following copied from comment:

```py
# Copied from transformers.models.bert.tokenization_bert.BasicTokenizer
class BasicTokenizer(object):
...
```

In the second case, the code of the class (and all its building blocks) should be copied and renamed to be prefixed by the model: for instance if you are copying code from the modeling_bert module to build Roberta, you replace all `BertLayer`, `BertOutput` etc... by `RobertaLayer`, `RobertaOutput`... 
You should then add a copied from statement (when the copy is without any modification) like this one:
```py
# Copied from transformers.models.bert.modeling_bert.BertAttention with Bert->Roberta
class RobertaAttention(nn.Module):
...
```
Note the replacement pattern that will adapt all names used. Note that:
- you can add more of those patterns, separated by a comma like [here](https://github.com/huggingface/transformers/blob/c28d04e9e252a1a099944e325685f14d242ecdcd/src/transformers/models/blenderbot_small/modeling_blenderbot_small.py#L1388).
- you can ask to replace all possible casings like [here](https://github.com/huggingface/transformers/blob/c28d04e9e252a1a099944e325685f14d242ecdcd/src/transformers/models/mobilebert/modeling_mobilebert.py#L1549)
- you can just copy one method and not the whole class like [here](https://github.com/huggingface/transformers/blob/c28d04e9e252a1a099944e325685f14d242ecdcd/src/transformers/models/roberta/modeling_roberta.py#L741)

**NB:** No need for copied from statements in the config (the defaults are probably different anyway).

## Objects to cover

### Configurations

- [x] Flaubert config (should not use XLM)
- [x] LayoutLM config (should not use Bert)
- [x] LongformerConfig (should not use Roberta)
- [x] MarkupLMConfig (should not Roberta)
- [x] RobertaConfig (should not use Bert)
- [x] XLM-ProphetNet config (should not use ProphetNet)
- [x] XLM-Roberta config (should not use Roberta)

### Models

- [x] BertGeneration (should not use BertEncoder)
- [x] Camembert (should not use Roberta) (PyTorch + TF)
- [x] Flaubert (should not use XLM) (PyTorch + TF)
- [ ] mT5: ~PyTorch~, TensorFlow, Flax (should not use T5)
- [x] XLM-ProphetNet (should not use ProphetNet)
- [ ] Xlm-Roberta: ~PyTorch~, TensorFlow, Flax (should not use Roberta)

### Tokenizers

- [x] BertJapanese (should not use any imports from tokenization bert)
- [x] Blenderbot (should not use Roberta) (slow/fast)
- [x] Clip (should not use BasicTokenizer from Bert)
- [x] ConvBERT (should not use Bert) (slow/fast)
- [x] Cpm tokenizer (should not use XLNet) (slow/fast)
- [x] Derberta tokenizer (should not use GPT2) (slow/fast)
- [x] DistilBert (should not use Bert) (slow/fast)
- [x] Electra (should not use Bert) (fast)
- [x] Flaubert (should not use XLM)
- [x] Funnel (should not use Bert) (slow/fast)
- [x] Herbert (should not BasicTokenizer from Bert and XLM)
- [x] LayoutLM (should not use Bert) (slow/fast)
- [x] LED (should not use BART) (slow/fast)
- [x] Longformer (should not use Roberta) (fast tokenizer)
- [x] Luke (should not use Roberta)
- [x] Lxmert (should not use Bert) (slow/fast)
- [x] MobileBert (should not use Bert) (slow/fast)
- [x] Openai-GPT (should not use BasicTokenizer from Bert)
- [x] ProphetNet (should not use BasicTokenzier and WordPieceTokenizer from Bert)
- [x] Retribert tokenizer (should not use Bert) (slow/fast)
- [x] Roformer tokenizer (should not use any imports from tokenization bert)
- [x] Squeezebert tokenizer (should not use Bert) (slow/fast)

Comments:
Hello! Happy to take LayoutLM Config and Tokenizer :) 

Hi, I would like to work on this. thanks.

Hi @OtherHorizon , as explained in the issue, please pick a model/config and/or tokenizer so that others know not to pick the same one :-)

Hi @sgugger, I like this philosophy, and I can work on DoRYing `LongformerConfig` and `Longformer Tokenizer` :) 

Hi @sgugger, I would like to work on RobertaConfig config and DistilBert tokenizer! 

Hi @sgugger, I would love to work on `Camembert` model.

Hello @sgugger I can work on the `Xlm-Roberta` model and config

Heya! I'd like to grab the `OpenAI-GPT` tokenizer :)

Hi! I'd Like to work on `Flaubert` config and model

Hi, I would like to work on mT5 and ProphetNet.

Hey! I can take a look at ClipTokenizer!

Hi, i would like to work on `Electra` and `Luke` Tokenizer

Hi, I would like to work on `BertGeneration`.


Hi! I'd Like to work on `ConvBERT` and `Lxmert` tokenizer.

Hi! I would like to work on `MobileBert` tokenizer.

Hello! I can also take up refactoring XLM-ProphetNet model and config. 

Hello, commenting to mark retribert

Hello! I would like to work on `Herbert` tokenizer.

Hello, I would like to work on Roformer tokenizer

I can work on LED 

Hi, I'd would gladly like to work on the Funnel tokenizer.

Hi all - I'd like to work on `Blenderbot` and `Squeezebert tokenizer` 😄

Hi, I'd like to work on `MarkupLM` config. 

Coming in to mark BertJapanese and Cpm now that I have some idea of how this goes

I would like to work on `Derberta tokenizer` maybe a typo . Just to confirm, task is to remove `GPT2Tokenizer` from https://github.com/huggingface/transformers/blob/bc21aaca789f1a366c05e8b5e111632944886393/src/transformers/models/deberta/tokenization_deberta.py#L66

I can work on `mt5 model` if free! @divyanshugit Can I take the issue, if you haven't already started?

hi @sgugger i would love to work on MarkupLMConfig

@sgugger Would like to work on the Roberta config.Please assign me

hi @sgugger i can try working on the ~~XLM-ProphetNet model~~ 
edit: just saw there was a pr for this, I can take whatever is left (if any)

The fast tokenizers for ELECTRA and Longformer are still available FYI :-)

@sgugger I would like to contribute fast tokenizers `ELECTRA` and `Longformer`.

Edit :
@sirmammingtonham  I missed your message. I can take `ELECTRA`, you can take `Longformer` ?

@sgugger I would like to work on `Funnel Tokenizer`

@Threepointone4 yep I can take the longformer fast tokenizer thanks!

@sgugger , Are there any tasks available further. I'd like to work on this


I think everything is taken as of now. If someone reports back they can't complete the model they picked, I will let you know!

@sgugger what if the code is changed in the source after it has been copied to the target? Do you have to manually copy again the new version?


No, that's what the `Copied from` statements are for, they will make sure the code always stay up to date.

Hi @sgugger  I would like to work on RobertaConfig 

Hi @sgugger , I would like to work on `BertJapanese` and `Cpm` tokenizer.

Hi @sgugger, while browsing through the different models, I realized that CamembertConfig is dependent on RobertaConfig. But, it seems the config is exactly the same as RobertaConfig. So essentially CamembertConfig is actually dependent on BertConfig, since RobertaConfig is dependent on BertConfig. 

However, I could not find CamembertConfig on the list above, so I am unsure if there is a need for seperating the RobertaConfig dependecy from CamembertConfig. Just incase it is, I will start a PR and maybe I can get some feedback from there.

Hi @sgugger, I would like to work on RobertaConfig next. Unsure if someone is working on it currently or not, since the last [pull request](https://github.com/huggingface/transformers/pull/19856) submitted for RobertaConfig was 14 days ago. That pull request was later closed, and no new PR was opened since then. I will submit a PR just incase, no one is working on it.

Hi @sgugger ! I would like to work on `Luke` tokenizer

Hey @sgugger, Can I take the `mT5` model and `DistilBert` tokenizer? Thanks,

Sure

mt5 is the last one on the list!

@sgugger I'll give mt5 a shot

Ah, @SD-13 are you still working on this?

> Ah, @SD-13 are you still working on this?

Yes, I am on it.

@SD-13 No worries! If you haven't started already, I've already been poking around at it for a bit and I'd like to keep going with it. Totally up to you, though :D

@SamuelzXu , I already started! Please feel free to take some other issue, thanks for asking though :)

Hi @SD-13 , I noticed for your PR you didn't make any modifications to the mt5 tokenizers, which have dependencies on T5Tokenizer. Mind if I take care of that part if you're not already?

Hey @SamuelzXu, I don't think it's there.


As @ydshieh discovered, a couple of models slipped through the cracks, check the list at the top if you still want to contribute :-) 

Hey @sgugger , Can I take the `Xlm-Roberta` and `mt5` model. Thanks,

@SD-13 You are very welcome to take them. But it would be a good idea to work on one model type at a time :-) and take another one if it's still available ❤️ when you finish the selected one.

@ydshieh , yeah very true, thanks for reminding that. I would take `mt5`

@SD-13 I was referring to [this line](https://github.com/huggingface/transformers/blob/4f1c9d162ee53004be4d63b76627bd0c2f5a31a9/src/transformers/models/mt5/__init__.py#L37) in init.py which defines MT5Tokenizer as T5Tokenizer

@SD-13 [And also](https://github.com/huggingface/transformers/blob/4f1c9d162ee53004be4d63b76627bd0c2f5a31a9/src/transformers/models/mt5/__init__.py#L44) MT5TokenizerFast defined as T5TokenizerFast

@sgugger I'll take a look at the conversion for XLM-Roberta. Could you add [MT5Tokenizer](https://github.com/huggingface/transformers/blob/4f1c9d162ee53004be4d63b76627bd0c2f5a31a9/src/transformers/models/mt5/__init__.py#L37) and [MT5TokenizerFast](https://github.com/huggingface/transformers/blob/4f1c9d162ee53004be4d63b76627bd0c2f5a31a9/src/transformers/models/mt5/__init__.py#L44) to the list please? Looks like they're defined as the T5Tokenizer and T5TokenizerFast, respectively.