TFDistilBertModelTest.test_pt_tf_model_equivalence thrown while merging after PR
Author: TheEdoardo93
Pull request: no
Labels: name: wontfix

## 🐛 Bug

I've seen that some PRs failed because of the same error, such as #2239 (today) and #2237 (today). If I'm not remember wrong, there is also another PR in the last days involved @rlouf that he had the same error. Even if the changes made not affect `DistilBertModel` such as #2237, this error occurs!
Question: **Is it a bug in Transformers or a bug in our code submitted?**

## To Reproduce

Steps to reproduce the behavior: after submitting a PR to Transformers library, it occurs
```
=================================== FAILURES ===================================
______________ TFDistilBertModelTest.test_pt_tf_model_equivalence ______________

self = <transformers.tests.modeling_tf_distilbert_test.TFDistilBertModelTest testMethod=test_pt_tf_model_equivalence>

    def test_pt_tf_model_equivalence(self):
        if not is_torch_available():
            return
    
        import torch
        import transformers
    
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
    
        for model_class in self.all_model_classes:
            pt_model_class_name = model_class.__name__[2:]  # Skip the "TF" at the beggining
            pt_model_class = getattr(transformers, pt_model_class_name)
    
            config.output_hidden_states = True
            tf_model = model_class(config)
            pt_model = pt_model_class(config)
    
            # Check we can load pt model in tf and vice-versa with model => model functions
            tf_model = transformers.load_pytorch_model_in_tf2_model(tf_model, pt_model, tf_inputs=inputs_dict)
            pt_model = transformers.load_tf2_model_in_pytorch_model(pt_model, tf_model)
    
            # Check predictions on first output (logits/hidden-states) are close enought given low-level computational differences
            pt_model.eval()
            pt_inputs_dict = dict((name, torch.from_numpy(key.numpy()).to(torch.long))
                                  for name, key in inputs_dict.items())
            with torch.no_grad():
                pto = pt_model(**pt_inputs_dict)
            tfo = tf_model(inputs_dict, training=False)
            tf_hidden_states = tfo[0].numpy()
            pt_hidden_states = pto[0].numpy()
            tf_hidden_states[np.isnan(tf_hidden_states)] = 0
            pt_hidden_states[np.isnan(pt_hidden_states)] = 0
            max_diff = np.amax(np.abs(tf_hidden_states - pt_hidden_states))
>           self.assertLessEqual(max_diff, 2e-2)
E           AssertionError: 3.107201 not less than or equal to 0.02

transformers/tests/modeling_tf_common_test.py:139: AssertionError
```

## Expected behavior

No error was thrown.

## Environment

* OS: **Ubuntu 16.04**
* Python version: **3.6.9**
* PyTorch version: **1.3.1**
* PyTorch Transformers version (or branch): **master**
* Using GPU ? **Indifferent**
* Distributed of parallel setup ? **Indifferent**
* Any other relevant information:

Comments:
I've seen the same error when running the test suite locally.

I know how to reproduce/debug this particular failure so I'll take a look on monday (unless someone beats me to it)

aaarg, I can't reproduce it locally anymore.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.