noamaanMulla-03/transformers-issues huggingface dataset:
Incorrect docstring for BertForMaskedLM
Author: Rocketknight1
Pull request: yes

The docstring for the head_mask argument to the BertForMaskedLM class is repeated and one is incorrect - I presume it's just a copy-paste mistake.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/718?src=pr&el=h1) Report
> Merging [#718](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/718?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/commit/98dc30b21e3df6528d0dd17f0910ffea12bc0f33?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/718/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/718?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #718   +/-   ##
=======================================
  Coverage   62.22%   62.22%           
=======================================
  Files          18       18           
  Lines        3979     3979           
=======================================
  Hits         2476     2476           
  Misses       1503     1503
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/718?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_pretrained\_bert/modeling.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/718/diff?src=pr&el=tree#diff-cHl0b3JjaF9wcmV0cmFpbmVkX2JlcnQvbW9kZWxpbmcucHk=) | `79.49% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/718?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/718?src=pr&el=footer). Last update [98dc30b...8d6a118](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/718?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks, we'll make a big clean up of the docstrings for the coming release.
Add tie_weights to XLNetForSequenceClassification
Author: Strideradu
Pull request: yes

XLNetForSequenceClassification doesn't have tie_weights() but initialization will call it, or we can made a function in XLNetPretrainedModel?

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/716?src=pr&el=h1) Report
> Merging [#716](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/716?src=pr&el=desc) into [xlnet](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/commit/c946bb51a61f67b0c9eaae1c9cf6f164a7748e37?src=pr&el=desc) will **increase** coverage by `0.03%`.
> The diff coverage is `50%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/716/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/716?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##            xlnet     #716      +/-   ##
==========================================
+ Coverage   62.18%   62.22%   +0.03%     
==========================================
  Files          22       22              
  Lines        4742     4744       +2     
==========================================
+ Hits         2949     2952       +3     
+ Misses       1793     1792       -1
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/716?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_pretrained\_bert/modeling\_xlnet.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/716/diff?src=pr&el=tree#diff-cHl0b3JjaF9wcmV0cmFpbmVkX2JlcnQvbW9kZWxpbmdfeGxuZXQucHk=) | `65.16% <50%> (-0.06%)` | :arrow_down: |
| [pytorch\_pretrained\_bert/tokenization\_gpt2.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/716/diff?src=pr&el=tree#diff-cHl0b3JjaF9wcmV0cmFpbmVkX2JlcnQvdG9rZW5pemF0aW9uX2dwdDIucHk=) | `83.51% <0%> (+1.06%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/716?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/716?src=pr&el=footer). Last update [c946bb5...00547bd](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/716?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks but we don't need that, the model was work in progress.
Update Adam optimizer to follow pytorch convention for betas parameter (#510)
Author: tonianelope
Pull request: yes

see #510

Update optimiser to follow pytorch convention ([Adam Optimiser](https://pytorch.org/docs/stable/optim.html#torch.optim.Adam)) instead of tensorflow, to allow for better integration with other pytorch libraries and frameworks.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/723?src=pr&el=h1) Report
> Merging [#723](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/723?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/commit/98dc30b21e3df6528d0dd17f0910ffea12bc0f33?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `83.33%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/723/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/723?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #723   +/-   ##
=======================================
  Coverage   62.22%   62.22%           
=======================================
  Files          18       18           
  Lines        3979     3979           
=======================================
  Hits         2476     2476           
  Misses       1503     1503
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/723?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_pretrained\_bert/optimization.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/723/diff?src=pr&el=tree#diff-cHl0b3JjaF9wcmV0cmFpbmVkX2JlcnQvb3B0aW1pemF0aW9uLnB5) | `74.26% <100%> (ø)` | :arrow_up: |
| [pytorch\_pretrained\_bert/optimization\_openai.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/723/diff?src=pr&el=tree#diff-cHl0b3JjaF9wcmV0cmFpbmVkX2JlcnQvb3B0aW1pemF0aW9uX29wZW5haS5weQ==) | `34.84% <66.66%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/723?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/723?src=pr&el=footer). Last update [98dc30b...c988590](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/723?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Ok, can you update the examples that use these optimizers as well?

> Ok, can you update the examples that use these optimizers as well?

I had a look at the examples, all seem to use the default values for b1/b2 so there shouldn't be any change required.

Perfect!
Adam optimiser not following Pytorch conventions
Author: tonianelope
Pull request: no
Labels: name: Help wanted

name: wontfix

Both [BertAdam](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/optimization.py) and [OpenAIAdam](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/optimization_openai.py) don't follow the pytroch convetion to define the `betas` parameter for  [Adam Optimisers](https://pytorch.org/docs/stable/optim.html#torch.optim.Adam) as a tuple, but instead has parameters `b1` and `b2`.

Pytorch based libraries like fastai expect the optimizer `betas` to be a tuple.
Any reason `b1/2` is used instead of a tuple? Would be great to change so the optimisers can integrate with other pytorch libraries.



Comments:
We could update that indeed, that's just a relic of the Tensorflow conversion.
Do you want to submit a PR? Otherwise I'll do it when I work on the next release.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Same loss values but different eval result
Author: a-maci
Pull request: no
Labels: name: Need more information

name: Discussion

name: wontfix

I am experimenting with low-precision on the pre-trained BERT for SQuAD scenario. 
I am seeing a strange issue: the loss value when fine-tuning the model with FP16 is very similar to the loss value when fine-tuning the model at Int8. However, the eval results are are quite different -- with Int8, the results are quite bad (f1 = 3) compared to f1=88 with FP16.

Any idea what is going on and suggestions for debugging?

Comments:
I have never tried Int8 in PyTorch.
Can you share some code so we can have a look?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How many epochs are necessary for finetuning BERT?
Author: search4mahesh
Pull request: no
Labels: name: Discussion

name: wontfix

Hi,
Could somebody provide some insights on how many epochs are necessary for finetuning bert model?
Google BERT has 100000 steps.(total_data/batch_size)

flags.DEFINE_integer("num_train_steps", 100000, "Number of training steps.") 

Thanks
Mahesh

Comments:
I have tried to finetune GPT rather than BERT. An appropriate running epochs is **3** in the generation setting, including learning on embedding of some custom special tokens. Hope it help you :)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
More SEPs
Author: shawnkx
Pull request: no
Labels: name: Discussion

name: wontfix

I want to segment input sentences in more segments, like [CLS]S1[SEP]S2[SEP]S3[SEP]. Therefore, when I convert example to features, I do the following.
`segment_ids = [0] * len(tokens_s1)`
`segment_ids += [1] * len(tokens_s2)`
`segment_ids += [2] * len(tokens_s2)`

but I got the following error when I run the `self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)`:

>   File "/home/xiangk/anaconda2/envs/pytorch0.4/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 1065, in forward
    sequence_output, _ = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
  File "/home/xiangk/anaconda2/envs/pytorch0.4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xiangk/anaconda2/envs/pytorch0.4/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 712, in forward
    embedding_output = self.embeddings(input_ids, token_type_ids)
  File "/home/xiangk/anaconda2/envs/pytorch0.4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xiangk/anaconda2/envs/pytorch0.4/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 264, in forward
    embeddings = self.dropout(embeddings)
  File "/home/xiangk/anaconda2/envs/pytorch0.4/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xiangk/anaconda2/envs/pytorch0.4/lib/python3.6/site-packages/torch/nn/modules/dropout.py", line 53, in forward
    return F.dropout(input, self.p, self.training, self.inplace)
  File "/home/xiangk/anaconda2/envs/pytorch0.4/lib/python3.6/site-packages/torch/nn/functional.py", line 595, in dropout
    return _functions.dropout.Dropout.apply(input, p, training, inplace)
  File "/home/xiangk/anaconda2/envs/pytorch0.4/lib/python3.6/site-packages/torch/nn/_functions/dropout.py", line 40, in forward
    ctx.noise.bernoulli_(1 - ctx.p).div_(1 - ctx.p)
RuntimeError: Creating MTGP constants failed. at /opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCTensorRandom.cu:34

after changing `segment_ids += [2] * len(tokens_s2)` to `segment_ids += [1] * len(tokens_s2)`, everything seems work but that is not what I want. Any suggestions? Thanks!



Comments:
Hi, only two segment labels are pre-trained in BERT.
You could fine-tune a new vocabulary token but we don't have a script to do that currently so you would have to modify the vocabulary and model.
GPT and GPT-2 have option to do that where you can take inspiration from.
I'm happy to welcome a PR on this if somebody feels like giving it a try.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
When not loading a pretrained model, all layers are initialized with copies of the same weights
Author: Rocketknight1
Pull request: yes

Although this repo is mostly used for loading and training pre-trained BERT models, the code does support model initialization too! However, I found an issue with the initialization code - because it just makes one layer and copies it, the weights will be identical across all layers at initialization. This probably isn't fatal, since they'll hopefully diverge over time, but it seems a bit odd and it isn't how the [Google BERT repo does it](https://github.com/google-research/bert/blob/master/modeling.py#L827-L882).

I replaced the copies with separate layer initializations instead, which should fix this problem.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/742?src=pr&el=h1) Report
> Merging [#742](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/742?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/commit/dad3c7a485b7ffc6fd2766f349e6ee845ecc2eee?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/742/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/742?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #742      +/-   ##
==========================================
- Coverage   62.27%   62.26%   -0.01%     
==========================================
  Files          18       18              
  Lines        3979     3978       -1     
==========================================
- Hits         2478     2477       -1     
  Misses       1501     1501
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/742?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_pretrained\_bert/modeling.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/742/diff?src=pr&el=tree#diff-cHl0b3JjaF9wcmV0cmFpbmVkX2JlcnQvbW9kZWxpbmcucHk=) | `79.46% <100%> (-0.04%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/742?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/742?src=pr&el=footer). Last update [dad3c7a...2c03c10](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/742?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Oops, my bad - I just realized you initialize weights in `BertModel` after creating them. Never mind!
GPT-2 fine tunning
Author: Jonbean
Pull request: no
Labels: name: wontfix

I wonder if GPT-2 model has some examples of how to do fine tuning like GPT. The DoubleHeadsModel interface of GPT-2 looks similar to GPT. But there's no special token handler for GPT-2 tokenizer. Is that necessary? 

Comments:
What kind of task are you fine-tuning on? I If it's something like ROCstories task, you need the extra tokens. I think people are doing BERT for up-stream tasks because birectional context gives better results than left-to-right

Hi @yaroslavvb, I am mostly focusing on classification tasks(like ROCstories as you mentioned). I just want to confirm that the special tokens used by GPT and GPT2 should be treated the same(add to the end of the vocabulary and feed in along with the original text). 

We would like to run both BERT and GPT2 which are the two SOTA models, since they shine in different ways. 

I will be happy to submit a pull request if this is a valid extension to the code base or if anyone is interested. 

Indeed we should probably add additional embeddings for GPT-2 also.
I'll give it a look, should be pretty easy to add.

I just tried implementing the changes suggested in order to make gpt-2 amenable to being finetuned on the Cloze Story Task from ROCStories. However, my eval-accuracy seems to be topping out at 68%. Is this what others are getting?

I also want to fine tune gpt-2 for qa and run it on squad. I am new to the field. Should I be following BertForQuestionAnswering and run BERT or SQuAD as a model to do the same for gpt-2? 

Hi @rohuns, I met the same situation, the performance of the pre-trained GPT-2 with extra task head is poor on both ROC and sts. I don't truly know the reason. 
My hypothesis is GPT-2 was not trained in the same "multi-task" fashion as the GPT1, therefore adding the special token will destroy the model or at least having hard time generating good representation for the sentence for the downstream tasks. 
I hope someone can get the fine-tuning work to disprove the above hypothesis. 


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Should I use weight_decay or weight_decay_rate?
Author: lemonhu
Pull request: no
Labels: name: wontfix

Thanks for the awesome work.

Just as line [simple_lm_finetuning.py#L540](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/lm_finetuning/simple_lm_finetuning.py#L540), When I use bert for downstream tasks, should I use `weight_decay` or `weight_decay_rate` when I add a decay operation to the training parameters?

What if I use apex for mixed precision training?

Comments:
According to the instructions [module-torch.optim](https://pytorch.org/docs/stable/optim.html?highlight=torch%20optim#module-torch.optim) from PyTorch API and [fused_adam.py](https://github.com/NVIDIA/apex/blob/master/apex/optimizers/fused_adam.py) from apex repo, I think `weight_decay` and `weight_decay_rate` are unified and unified into `weight_decay`, is it correct to understand?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
extending of Transformer-XL for new tasks 
Author: cherepanovic
Pull request: no
Labels: name: wontfix

Hello community,

I am looking for an example which could help me to extend the Transformer XL to a model similar to bert-as-service model [1]. I would like to know how to set up new layers on the pretrained Tranformer XL and train the last new layers or the whole model. Could anyone give me an advice regarding this issue. Thank a lot

[1] - https://github.com/hanxiao/bert-as-service

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
__init__() got an unexpected keyword argument 'do_basic_tokenize'
Author: lcswillems
Pull request: no
Labels: name: Need more information

name: wontfix

In the README, this line is written:

```
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True, do_basic_tokenize=True)
```

But when I execute it, I get this error:

```
__init__() got an unexpected keyword argument 'do_basic_tokenize'
```



Comments:
Which version of pytorch-pretrained-bert are you using?
Can you give the full error message to see which call to `__init__()` is failing?
We should have the keyword argument [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/3d78e226e68a5c5d0ef612132b601024c3534e38/pytorch_pretrained_bert/tokenization.py#L77) 

I have the last version (0.6.1).

This is what I have on my computer:

![image](https://user-images.githubusercontent.com/5437552/56668476-db88db00-66af-11e9-92a1-5704ae4c157e.png)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Cannot reproduce results from version 0.4.0
Author: hguan6
Pull request: no

Hi, I have a research project that I did a few months ago. Now I have problem reproducing results of 0.4.0, and unfortunately, I lost version 0.4.0. Can you please send me the code of this version to hguan6@asu.edu? In fact, I am not quite sure it's 0.4.0, but I remember I did it in March 2019.

Comments:
`pip install pytorch-pretrained-bert==0.4.0` should work normally

Though if you did it with the latest release in March 2019 it was probably more 0.6.1 (see the list and dates here: https://github.com/huggingface/pytorch-pretrained-BERT/releases) so `pip install pytorch-pretrained-bert==0.6.1`

Thank you!

On Sun, Jun 30, 2019, 08:40 Thomas Wolf <notifications@github.com> wrote:

> Though if you did it with the latest release in March 2019 it was probably
> more 0.6.1 (see the list and dates here:
> https://github.com/huggingface/pytorch-pretrained-BERT/releases) so pip
> install pytorch-pretrained-bert==0.6.1
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/pytorch-pretrained-BERT/issues/743?email_source=notifications&email_token=AEX53CZ4RLMCBSQFO2445QDP5DAWZA5CNFSM4H4MVPT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY4NP4Y#issuecomment-507041779>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AEX53C7IV2A7DRL7723V573P5DAWZANCNFSM4H4MVPTQ>
> .
>
[Dropout] why  there is no dropout for the dev and eval?
Author: Albert-Ma
Pull request: no

I do not see any dropout layer after `get_pooled_output()`  in the tf version referred to [here](https://github.com/google-research/bert/blob/master/run_classifier.py#L590). Why do you add a dropout layer in your implemention?

Comments:
@thomwolf  Thanks!

line 604

> line 604

Thanks so much. my mistake.

Do you know why there is no dropout for the dev and eval?


first of all, no one uses dropout at evaluation stage as it's a regularizer. The difference of implementation is due to the fact that a dropout layer in pytorch behaves differently once you turn on a evaluation model (i.e., model.eval()). So even though it has a dropout layer it doesn't take any effect (p=0) at evaluation.

> first of all, no one uses dropout at evaluation stage as it's a regularizer. The difference of implementation is due to the fact that a dropout layer in pytorch behaves differently once you turn on a evaluation model (i.e., model.eval()). So even though it has a dropout layer it doesn't take any effect (p=0) at evaluation.

Thanks for help me clarify this problem.