Fix FFN dropout in TFAlbertLayer, and split dropout in TFAlbertAttent…
Author: jarednielsen
Pull request: yes

…ion into two separate dropout layers.

1) The dropout in TFAlbertLayer's FFN submodule was never used, since it was applied to hidden_states, which was immediately overwritten. Fixed to apply to ffn_output.

2) Likewise, the dropout in TFAlbertAttention defaulted to the dropout defined in TFBertAttention. The ALBERT paper handles this a bit differently, instead having two separate parameters controlling dropout probabilities. See https://github.com/google-research/albert/blob/master/modeling.py#L971-L993 for an example of how it is coded in the original repo. The `attention_1` scope uses `attention_probs_dropout_prob`, while the `output` scope uses `hidden_dropout_prob`.

Since the default dropout probabilities are 0 for ALBERT, these changes shouldn't affect the model accuracies when using default config. It will help practitioners and researchers who are hyperparameter tuning to have proper dropout implementation.

Comments:
@LysandreJik Any thoughts?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Not stale

Will take care of the conflict and merge after https://github.com/huggingface/transformers/pull/6247 is merged.

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4323?src=pr&el=h1) Report
> Merging [#4323](https://codecov.io/gh/huggingface/transformers/pull/4323?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/155288f04ba9a5d0a0e4d5be4f6d4e808ad8cfff&el=desc) will **decrease** coverage by `2.57%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4323/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4323?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4323      +/-   ##
==========================================
- Coverage   79.94%   77.37%   -2.58%     
==========================================
  Files         153      153              
  Lines       27902    27907       +5     
==========================================
- Hits        22307    21593     -714     
- Misses       5595     6314     +719     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4323?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/4323/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hbGJlcnQucHk=) | `82.15% <100.00%> (+0.10%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/4323/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9hbGJlcnQucHk=) | `88.39% <100.00%> (+0.04%)` | :arrow_up: |
| [src/transformers/optimization.py](https://codecov.io/gh/huggingface/transformers/pull/4323/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9vcHRpbWl6YXRpb24ucHk=) | `25.55% <0.00%> (-70.00%)` | :arrow_down: |
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/4323/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `26.98% <0.00%> (-52.81%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/4323/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `34.11% <0.00%> (-30.36%)` | :arrow_down: |
| [src/transformers/optimization\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/4323/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9vcHRpbWl6YXRpb25fdGYucHk=) | `33.33% <0.00%> (-24.33%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/4323/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9hdXRvLnB5) | `48.79% <0.00%> (-18.08%)` | :arrow_down: |
| [src/transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/4323/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hdXRvLnB5) | `64.16% <0.00%> (-14.46%)` | :arrow_down: |
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/4323/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `13.76% <0.00%> (-14.38%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/4323/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9ncHQyLnB5) | `65.68% <0.00%> (-6.16%)` | :arrow_down: |
| ... and [12 more](https://codecov.io/gh/huggingface/transformers/pull/4323/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4323?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4323?src=pr&el=footer). Last update [155288f...a6da32a](https://codecov.io/gh/huggingface/transformers/pull/4323?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, just merged with master and fixed conflicts.

Thanks a lot for your contribution @jarednielsen !