Repository: huggingface/transformers

Issue:
Call to torch.pow() passing integer as exponent isn't per PyTorch docs
# 🐛 Bug

## Information

I am running the GPT2 pytest and it fails to load in [PyTorch Glow](https://github.com/pytorch/glow) because the model calls _torch.pow()_ with an integer for the _exponent_ parameter.

Per the PyTorch documentation (https://pytorch.org/docs/master/torch.html?highlight=torch%20pow#torch.pow):

> exponent can be either a single float number or a Tensor with the same number of elements as input.

and

>exponent (float or tensor) – the exponent value

The test was run with the following modifications to enable Glow:

```
diff --git a/src/transformers/modeling_gpt2.py b/src/transformers/modeling_gpt2.py
index 12013996..d6f39007 100644
--- a/src/transformers/modeling_gpt2.py
+++ b/src/transformers/modeling_gpt2.py
@@ -265,7 +265,7 @@ class GPT2PreTrainedModel(PreTrainedModel):
         if isinstance(module, (nn.Linear, nn.Embedding, Conv1D)):
             # Slightly different from the TF version which uses truncated_normal for initialization
             # cf https://github.com/pytorch/pytorch/pull/5617
-            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+            module.weight.data.normal_(mean=0.0, std=0.02) #self.config.initializer_range)
             if isinstance(module, (nn.Linear, Conv1D)) and module.bias is not None:
                 module.bias.data.zero_()
         elif isinstance(module, nn.LayerNorm):
diff --git a/tests/test_modeling_common.py b/tests/test_modeling_common.py
index 1d11ef8c..9df209f7 100644
--- a/tests/test_modeling_common.py
+++ b/tests/test_modeling_common.py
@@ -27,6 +27,7 @@ from .utils import require_torch, slow, torch_device
 
 if is_torch_available():
     import torch
+    import torch_glow
     import numpy as np
 
     from transformers import (
@@ -209,6 +210,8 @@ class ModelTesterMixin:
             inputs = inputs_dict["input_ids"]  # Let's keep only input_ids
 
             try:
+                torch_glow.enableFusionPass()
+                torch_glow.setGlowBackend('Interpreter')
                 traced_gpt2 = torch.jit.trace(model, inputs)
             except RuntimeError:
                 self.fail("Couldn't trace module.")
```

## To reproduce

Steps to reproduce the behavior:

1. python -m pytest -v -k 'test_torchscript and not test_torchscript_' ./tests/test_modeling_gpt2.py

## Expected behavior

Expect exponent to be passed as a float per the documentation so that model loaders adhering to the docs will be able to load the model.

## Environment info
     
- `transformers` version: 2.8.0                                                                                                                                                                                                         
- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-centos-7.5.1804-Core
- Python version: 3.6.8                                                                                                                                                                                                                 
- PyTorch version (GPU?): 1.5.0a0+8eaafbd (False)
- Tensorflow version (GPU?): not installed (NA)                                                                                                                                                                                         
- Using GPU in script?: no                                                                                  
- Using distributed or parallel set-up in script?: no 

Call to torch.pow() passing integer as exponent isn't per PyTorch docs
# 🐛 Bug

## Information

I am running the GPT2 pytest and it fails to load in [PyTorch Glow](https://github.com/pytorch/glow) because the model calls _torch.pow()_ with an integer for the _exponent_ parameter.

Per the PyTorch documentation (https://pytorch.org/docs/master/torch.html?highlight=torch%20pow#torch.pow):

> exponent can be either a single float number or a Tensor with the same number of elements as input.

and

>exponent (float or tensor) – the exponent value

The test was run with the following modifications to enable Glow:

```
diff --git a/src/transformers/modeling_gpt2.py b/src/transformers/modeling_gpt2.py
index 12013996..d6f39007 100644
--- a/src/transformers/modeling_gpt2.py
+++ b/src/transformers/modeling_gpt2.py
@@ -265,7 +265,7 @@ class GPT2PreTrainedModel(PreTrainedModel):
         if isinstance(module, (nn.Linear, nn.Embedding, Conv1D)):
             # Slightly different from the TF version which uses truncated_normal for initialization
             # cf https://github.com/pytorch/pytorch/pull/5617
-            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+            module.weight.data.normal_(mean=0.0, std=0.02) #self.config.initializer_range)
             if isinstance(module, (nn.Linear, Conv1D)) and module.bias is not None:
                 module.bias.data.zero_()
         elif isinstance(module, nn.LayerNorm):
diff --git a/tests/test_modeling_common.py b/tests/test_modeling_common.py
index 1d11ef8c..9df209f7 100644
--- a/tests/test_modeling_common.py
+++ b/tests/test_modeling_common.py
@@ -27,6 +27,7 @@ from .utils import require_torch, slow, torch_device
 
 if is_torch_available():
     import torch
+    import torch_glow
     import numpy as np
 
     from transformers import (
@@ -209,6 +210,8 @@ class ModelTesterMixin:
             inputs = inputs_dict["input_ids"]  # Let's keep only input_ids
 
             try:
+                torch_glow.enableFusionPass()
+                torch_glow.setGlowBackend('Interpreter')
                 traced_gpt2 = torch.jit.trace(model, inputs)
             except RuntimeError:
                 self.fail("Couldn't trace module.")
```

## To reproduce

Steps to reproduce the behavior:

1. python -m pytest -v -k 'test_torchscript and not test_torchscript_' ./tests/test_modeling_gpt2.py

## Expected behavior

Expect exponent to be passed as a float per the documentation so that model loaders adhering to the docs will be able to load the model.

## Environment info
     
- `transformers` version: 2.8.0                                                                                                                                                                                                         
- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-centos-7.5.1804-Core
- Python version: 3.6.8                                                                                                                                                                                                                 
- PyTorch version (GPU?): 1.5.0a0+8eaafbd (False)
- Tensorflow version (GPU?): not installed (NA)                                                                                                                                                                                         
- Using GPU in script?: no                                                                                  
- Using distributed or parallel set-up in script?: no 



Discussion:
I think you are correct. Any way you'd open a PR? Otherwise we'll get on it in the next few days/weeks
I think you are correct. Any way you'd open a PR? Otherwise we'll get on it in the next few days/weeks

Resolving patch:
diff --git a/src/transformers/activations.py b/src/transformers/activations.py
--- a/src/transformers/activations.py
+++ b/src/transformers/activations.py
@@ -26,7 +26,7 @@ def gelu_new(x):
     """ Implementation of the gelu activation function currently in Google Bert repo (identical to OpenAI GPT).
         Also see https://arxiv.org/abs/1606.08415
     """
-    return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
+    return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3.0))))
 
 
 if torch.__version__ < "1.4.0":


Test patch: