Add DeBERTa model
Author: BigBird01
Pull request: yes
Labels: name: model card

name: New model

Add DeBERTa model to hf transformers. DeBERTa applies two techniques to improve RoBERTa, one is disentangled attention, the other is enhanced mask decoder.  With 80GB training data, DeBERTa outperform RoBERTa on a majority of NLU tasks, e.g. SQUAD, MNLI and RACE. Paper link: https://arxiv.org/abs/2006.03654 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5929?src=pr&el=h1) Report
> Merging [#5929](https://codecov.io/gh/huggingface/transformers/pull/5929?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/1fc4de69ed024e18b88cb6f040021630599de2f7?el=desc) will **decrease** coverage by `0.30%`.
> The diff coverage is `73.13%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5929/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5929?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5929      +/-   ##
==========================================
- Coverage   79.35%   79.05%   -0.31%     
==========================================
  Files         181      184       +3     
  Lines       35800    36660     +860     
==========================================
+ Hits        28410    28982     +572     
- Misses       7390     7678     +288     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5929?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/activations.py](https://codecov.io/gh/huggingface/transformers/pull/5929/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9hY3RpdmF0aW9ucy5weQ==) | `76.92% <50.00%> (-6.42%)` | :arrow_down: |
| [src/transformers/tokenization\_deberta.py](https://codecov.io/gh/huggingface/transformers/pull/5929/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fZGViZXJ0YS5weQ==) | `69.76% <69.76%> (ø)` | |
| [src/transformers/modeling\_deberta.py](https://codecov.io/gh/huggingface/transformers/pull/5929/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19kZWJlcnRhLnB5) | `73.26% <73.26%> (ø)` | |
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/5929/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.39% <100.00%> (+0.01%)` | :arrow_up: |
| [src/transformers/configuration\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/5929/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2F1dG8ucHk=) | `96.34% <100.00%> (+0.04%)` | :arrow_up: |
| [src/transformers/configuration\_deberta.py](https://codecov.io/gh/huggingface/transformers/pull/5929/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2RlYmVydGEucHk=) | `100.00% <100.00%> (ø)` | |
| [src/transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/5929/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hdXRvLnB5) | `87.11% <100.00%> (+0.06%)` | :arrow_up: |
| [src/transformers/tokenization\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/5929/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYXV0by5weQ==) | `92.64% <100.00%> (+0.10%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_lxmert.py](https://codecov.io/gh/huggingface/transformers/pull/5929/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9seG1lcnQucHk=) | `22.14% <0.00%> (-72.41%)` | :arrow_down: |
| [src/transformers/tokenization\_pegasus.py](https://codecov.io/gh/huggingface/transformers/pull/5929/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fcGVnYXN1cy5weQ==) | `46.03% <0.00%> (-49.21%)` | :arrow_down: |
| ... and [17 more](https://codecov.io/gh/huggingface/transformers/pull/5929/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5929?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5929?src=pr&el=footer). Last update [1fc4de6...0a08565](https://codecov.io/gh/huggingface/transformers/pull/5929?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Related issue #4858 

Someone is waiting for fine-tuning a new model :)

Very cool, looking forward to that model!!

Hello, May I know when will the PR be merged?



@BigBird01 it will probably take between 1 and 3 weeks to merge. 2,500 lines is a lot to review :) 
I made some comments, and can take another pass on this when they're addressed.


> @BigBird01 it will probably take between 1 and 3 weeks to merge. 2,500 lines is a lot to review :)
> I made some comments, and can take another pass on this when they're addressed.

Thanks!

Thanks for addressing the comments! Will take a look in a few days.

> Great, it's nearly done! Thanks a lot for your work on it.
> 
> What's left to do is:
> 
> * Ensure that the documentation is in the correct format
> * Enable the remaining tests
> 
> If you don't have time to work on it right now, let me know and I'll finish the implementation and merge it. Thanks!

@LysandreJik Thanks for the comments. It will be great if you can work on the rest:) Feel free to let me know if you have any questions on the implementation. 


Okay @BigBird01, I have the PyTorch version ready, passing all tests and the docs cleaned up as well. Should I push directly on your branch, or do you want me to open a PR on your fork so that you can check my changes before applying them?

Before we merge there will be one final item we'll have to take care of, that's integration tests! It's necessary to ensure we don't diverge from the original implementation. [Here's an example with RoBERTa.](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_roberta.py#L397-L448)

Given that you have the original implementation, could you implement such tests? Thanks a lot!

> Okay @BigBird01, I have the PyTorch version ready, passing all tests and the docs cleaned up as well. Should I push directly on your branch, or do you want me to open a PR on your fork so that you can check my changes before applying them?

@LysandreJik  I just added you to the repo contributor. Please open a PR on it and I will merge your change into it then add integration tests. Thanks!


Hi @BigBird01, I just opened the pull request [here](https://github.com/BigBird01/transformers/pull/1).

Hi @BigBird01, did you get a chance to take a look at the PR?

> Hi @BigBird01, did you get a chance to take a look at the PR?

Sorry for the late reply. I just merged your changes and will try to add final test today. 

> Hi @BigBird01, did you get a chance to take a look at the PR?

@LysandreJik I just finished the final test. But I hit a isort error after I pushed the code to the repo, but the tests get passed on my local node. Could you help to take a look at it?

> Hi, left a last few comments and questions. Let me know if you do not have time to implement/answer these last changes and I'll do the last updates and merge `DeBERTa`.

@LysandreJik Thanks for the following up. I just replied most of your comments. Please feel free to make the last changes to merge the PR. Thanks in advance:)

I don't really understand the tracing changes, as the model does not pass the TorchScript tests. I'm removing this part, feel free to open a PR to add it back and set the `test_torchscript` flag to `True` in `test_modeling_deberta.py`. FYI, the error is the following:

```
    def save(self, *args, **kwargs):
        r"""
        save(f, _extra_files=ExtraFilesMap{})
    
        See :func:`torch.jit.save <torch.jit.save>` for details.
        """
>       return self._c.save(*args, **kwargs)
E       RuntimeError: 
E       Could not export Python function call 'XSoftmax'. Remove calls to Python functions before export. Did you forget add @script or @script_method annotation? If this is a nn.ModuleList, add it to __constants__:

```

There is an issue with the checkpoints uploaded on `huggingface.co`, as the base model identifier is `bert`, whereas this has been changed to `deberta`. This means that no weights are loaded on the model, as the base prefix doesn't fit.

Do you mind if I update the weights on S3 with so that the state dict changes from this:

```
[...]
'bert.embeddings.word_embeddings.weight', 
'bert.embeddings.position_embeddings.weight', 
'bert.embeddings.LayerNorm.weight',
'bert.embeddings.LayerNorm.bias', 
'bert.encoder.layer.0.attention.self.q_bias', 
'bert.encoder.layer.0.attention.self.v_bias'
[...]
```
to this?
```
[...]
'deberta.embeddings.word_embeddings.weight', 
'deberta.embeddings.position_embeddings.weight', 
'deberta.embeddings.LayerNorm.weight',
'deberta.embeddings.LayerNorm.bias', 
'deberta.encoder.layer.0.attention.self.q_bias', 
'deberta.encoder.layer.0.attention.self.v_bias'
[...]
```

> I don't really understand the tracing changes, as the model does not pass the TorchScript tests. I'm removing this part, feel free to open a PR to add it back and set the `test_torchscript` flag to `True` in `test_modeling_deberta.py`. FYI, the error is the following:
> 
> ```
>     def save(self, *args, **kwargs):
>         r"""
>         save(f, _extra_files=ExtraFilesMap{})
>     
>         See :func:`torch.jit.save <torch.jit.save>` for details.
>         """
> >       return self._c.save(*args, **kwargs)
> E       RuntimeError: 
> E       Could not export Python function call 'XSoftmax'. Remove calls to Python functions before export. Did you forget add @script or @script_method annotation? If this is a nn.ModuleList, add it to __constants__:
> ```

Sure. Let's remove it first and I can try to fix it later in a separate PR.

> There is an issue with the checkpoints uploaded on `huggingface.co`, as the base model identifier is `bert`, whereas this has been changed to `deberta`. This means that no weights are loaded on the model, as the base prefix doesn't fit.
> 
> Do you mind if I update the weights on S3 with so that the state dict changes from this:
> 
> ```
> [...]
> 'bert.embeddings.word_embeddings.weight', 
> 'bert.embeddings.position_embeddings.weight', 
> 'bert.embeddings.LayerNorm.weight',
> 'bert.embeddings.LayerNorm.bias', 
> 'bert.encoder.layer.0.attention.self.q_bias', 
> 'bert.encoder.layer.0.attention.self.v_bias'
> [...]
> ```
> 
> to this?
> 
> ```
> [...]
> 'deberta.embeddings.word_embeddings.weight', 
> 'deberta.embeddings.position_embeddings.weight', 
> 'deberta.embeddings.LayerNorm.weight',
> 'deberta.embeddings.LayerNorm.bias', 
> 'deberta.encoder.layer.0.attention.self.q_bias', 
> 'deberta.encoder.layer.0.attention.self.v_bias'
> [...]
> ```

Sure.

Okay @BigBird01, everything seems good to go, this should be merged very soon :)

Just one last question, for your models on the hub (`microsoft/deberta-base` and `microsoft/deberta-large`) in your configuration there is `position_biased_input` set to `False`, which means that in the embedding layer, the `position_embeddings` will be set to `None`:

```py
        if not self.position_biased_input:
            self.position_embeddings = None
```

However, in the model state dicts in the `pytorch_model.bin`, there is a layer containing the `position_embeddings`. What is correct here? Should there be `position_biased_input = True` in the configurations, or should this layer be removed from the state dicts? 

Thanks!

> Okay @BigBird01, everything seems good to go, this should be merged very soon :)
> 
> Just one last question, for your models on the hub (`microsoft/deberta-base` and `microsoft/deberta-large`) in your configuration there is `position_biased_input` set to `False`, which means that in the embedding layer, the `position_embeddings` will be set to `None`:
> 
> ```python
>         if not self.position_biased_input:
>             self.position_embeddings = None
> ```
> 
> However, in the model state dicts in the `pytorch_model.bin`, there is a layer containing the `position_embeddings`. What is correct here? Should there be `position_biased_input = True` in the configurations, or should this layer be removed from the state dicts?
> 
> Thanks!

Yes. It's used in the mask decoding part(EMD). We are still polishing that part and will release it once ready.

Thanks @patrickvonplaten, @sgugger for the reviews. Will implement the changes tonight.

Thanks for your work on this @BigBird01 :)

> Thanks for your work on this @BigBird01 :)

Thank you all to merge the code into master @LysandreJik  @patrickvonplaten 
One question, why after the merge we can't find the document of deberta model at https://huggingface.co/transformers/
Could you help to check that?

The documentation is online, you just have to click on `master` on the top left right under the hugging face logo as is done here: https://huggingface.co/transformers/master/. The next release will then show deberta docs as a default :-) 

@BigBird01, two slow tests are failing with the DeBERTa models. Could you show how you implemented the integration tests so that I may investigate?

> @BigBird01, two slow tests are failing with the DeBERTa models. Could you show how you implemented the integration tests so that I may investigate?

@LysandreJik  In the integration tests, I just feed the model with a fake input data and verify the output of the model. It's similar to RoBERTa tests. I may take a took at it today. 

Thanks! The DeBERTa may not be working correctly right now, knowing the source of the issue would be great. 

> Thanks! The DeBERTa may not be working correctly right now, knowing the source of the issue would be great.

The issue is due to the model failed to be loaded due to parameter name mismatch. It needs to update the model by changing the encoder name from 'bert' to 'deberta'.

> Thanks! The DeBERTa may not be working correctly right now, knowing the source of the issue would be great.

@LysandreJik  I just made a fix to the test failure. https://github.com/huggingface/transformers/pull/7645
Could you take a look?