Repository: huggingface/transformers

Issue:
torchscript tests fail with RuntimeError: normal_ expects std > 0.0, but found std=0
# 🐛 Bug

## Information

I am running the gpt2 torchscript test from master and a call to _normal_()_ fails because the _std_ parameter is zero. The error is not limited to the GPT2 model.


## To reproduce

Steps to reproduce the behavior:

1. python -m pytest -v -k 'test_torchscript and not test_torchscript_' ./tests/test_modeling_gpt2.py

The test fails with the following errors:

```
$ python -m pytest -v -k 'test_torchscript and not test_torchscript_' ./tests/test_m
odeling_gpt2.py
========================================================================================================= test session starts ==========================================================================================================
platform linux -- Python 3.6.8, pytest-5.2.0, py-1.8.1, pluggy-0.13.1 -- /local/mneilly/sw-platform-cawg/build/install-staging/sw-platform-sysroot/bin/python
cachedir: .pytest_cache
metadata: {'Python': '3.6.8', 'Platform': 'Linux-3.10.0-957.el7.x86_64-x86_64-with-centos-7.5.1804-Core', 'Packages': {'pytest': '5.2.0', 'py': '1.8.1', 'pluggy': '0.13.1'}, 'Plugins': {'forked': '1.1.3', 'html': '2.0.0', 'metadata'
: '1.8.0', 'xdist': '1.30.0'}}
rootdir: /local/mneilly/sw-platform-cawg/build/cawg-regression/pytorch-models/transformers/transformers/src/transformers
plugins: forked-1.1.3, html-2.0.0, metadata-1.8.0, xdist-1.30.0
collected 29 items / 28 deselected / 1 selected                                                                                                                                                                                        

tests/test_modeling_gpt2.py::GPT2ModelTest::test_torchscript FAILED                                                                                                                                                              [100%]

=============================================================================================================== FAILURES ===============================================================================================================
____________________________________________________________________________________________________ GPT2ModelTest.test_torchscript ____________________________________________________________________________________________________

self = <tests.test_modeling_gpt2.GPT2ModelTest testMethod=test_torchscript>

    def test_torchscript(self):
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
    
>       self._create_and_check_torchscript(config, inputs_dict)

tests/test_modeling_common.py:186: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_modeling_common.py:207: in _create_and_check_torchscript
    model = model_class(config=configs_no_init)
src/transformers/modeling_gpt2.py:353: in __init__
    self.init_weights()
src/transformers/modeling_utils.py:392: in init_weights
    self.apply(self._init_weights)
../../../../../../install-staging/sw-platform-sysroot/lib/python3.6/site-packages/torch/nn/modules/module.py:289: in apply
    module.apply(fn)
../../../../../../install-staging/sw-platform-sysroot/lib/python3.6/site-packages/torch/nn/modules/module.py:290: in apply
    fn(self)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = GPT2Model(
  (wte): Embedding(99, 32)
  (wpe): Embedding(512, 32)
  (drop): Dropout(p=0.1, inplace=False)
  (h): Modul...pout): Dropout(p=0.1, inplace=False)
      )
    )
  )
  (ln_f): LayerNorm((32,), eps=1e-05, elementwise_affine=True)
)
module = Embedding(99, 32)

    def _init_weights(self, module):
        """ Initialize the weights.
        """
        if isinstance(module, (nn.Linear, nn.Embedding, Conv1D)):
            # Slightly different from the TF version which uses truncated_normal for initialization
            # cf https://github.com/pytorch/pytorch/pull/5617
>           module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
E           RuntimeError: normal_ expects std > 0.0, but found std=0

src/transformers/modeling_gpt2.py:268: RuntimeError
=================================================================================================== 1 failed, 28 deselected in 1.97s ===================================================================================================
```

The test passes with the following modification:

```
diff --git a/src/transformers/modeling_gpt2.py b/src/transformers/modeling_gpt2.py
index 12013996..d6f39007 100644
--- a/src/transformers/modeling_gpt2.py
+++ b/src/transformers/modeling_gpt2.py
@@ -265,7 +265,7 @@ class GPT2PreTrainedModel(PreTrainedModel):
         if isinstance(module, (nn.Linear, nn.Embedding, Conv1D)):
             # Slightly different from the TF version which uses truncated_normal for initialization
             # cf https://github.com/pytorch/pytorch/pull/5617
-            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+            module.weight.data.normal_(mean=0.0, std=0.02) #self.config.initializer_range)
             if isinstance(module, (nn.Linear, Conv1D)) and module.bias is not None:
                 module.bias.data.zero_()
         elif isinstance(module, nn.LayerNorm):
``` 

Producing the following output:

```
$ python -m pytest -v -k 'test_torchscript and not test_torchscript_' ./tests/test_modeling_gpt2.py
========================================================================================================= test session starts ==========================================================================================================
platform linux -- Python 3.6.8, pytest-5.2.0, py-1.8.1, pluggy-0.13.1 -- /local/mneilly/sw-platform-cawg/build/install-staging/sw-platform-sysroot/bin/python
cachedir: .pytest_cache
metadata: {'Python': '3.6.8', 'Platform': 'Linux-3.10.0-957.el7.x86_64-x86_64-with-centos-7.5.1804-Core', 'Packages': {'pytest': '5.2.0', 'py': '1.8.1', 'pluggy': '0.13.1'}, 'Plugins': {'forked': '1.1.3', 'html': '2.0.0', 'metadata': '1.8.0', 'xdist': '1.30.0'}}
rootdir: /local/mneilly/sw-platform-cawg/build/cawg-regression/pytorch-models/transformers/transformers/src/transformers
plugins: forked-1.1.3, html-2.0.0, metadata-1.8.0, xdist-1.30.0
collected 29 items / 28 deselected / 1 selected                                                                                                                                                                                        

tests/test_modeling_gpt2.py::GPT2ModelTest::test_torchscript PASSED                                                                                                                                                              [100%]

=========================================================================================================== warnings summary ===========================================================================================================
tests/test_modeling_gpt2.py::GPT2ModelTest::test_torchscript
  /local/mneilly/sw-platform-cawg/build/cawg-regression/pytorch-models/transformers/transformers/src/transformers/src/transformers/modeling_gpt2.py:146: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
    w = w / math.sqrt(v.size(-1))

tests/test_modeling_gpt2.py::GPT2ModelTest::test_torchscript
  /local/mneilly/sw-platform-cawg/build/cawg-regression/pytorch-models/transformers/transformers/src/transformers/src/transformers/modeling_gpt2.py:148: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
    mask = self.bias[:, :, ns - nd : ns, :ns]

-- Docs: https://docs.pytest.org/en/latest/warnings.html

```
## Expected behavior

Expected test to pass

## Environment info

- `transformers` version: 2.8.0
- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-centos-7.5.1804-Core
- Python version: 3.6.8
- PyTorch version (GPU?): 1.5.0a0+8eaafbd (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -67,7 +67,7 @@
 extras["sklearn"] = ["scikit-learn"]
 extras["tf"] = ["tensorflow"]
 extras["tf-cpu"] = ["tensorflow-cpu"]
-extras["torch"] = ["torch==1.4.0"]
+extras["torch"] = ["torch"]
 
 extras["serving"] = ["pydantic", "uvicorn", "fastapi", "starlette"]
 extras["all"] = extras["serving"] + ["tensorflow", "torch"]


Test patch:
diff --git a/tests/test_modeling_common.py b/tests/test_modeling_common.py
--- a/tests/test_modeling_common.py
+++ b/tests/test_modeling_common.py
@@ -44,7 +44,7 @@ def _config_zero_init(config):
     configs_no_init = copy.deepcopy(config)
     for key in configs_no_init.__dict__.keys():
         if "_range" in key or "_std" in key or "initializer_factor" in key:
-            setattr(configs_no_init, key, 0.0)
+            setattr(configs_no_init, key, 1e-10)
     return configs_no_init
 
 
@@ -95,7 +95,7 @@ def test_initialization(self):
             for name, param in model.named_parameters():
                 if param.requires_grad:
                     self.assertIn(
-                        param.data.mean().item(),
+                        ((param.data.mean() * 1e9).round() / 1e9).item(),
                         [0.0, 1.0],
                         msg="Parameter {} of model {} seems not properly initialized".format(name, model_class),
                     )