loss.backward() TypeError seed issue for pretrained reformer model
Author: cnedwards
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.2
- Platform: Google Colab
- Python version:  3.7.10
- PyTorch version (GPU?): 1.7.0+cu101
- Using GPU in script?: no, but error also occurs on GPU
- Using distributed or parallel set-up in script?: no

### Who can help

@patrickvonplaten 

## Information

Model I am using: ReformerForSequenceClassification

The problem arises when using: Pretrained model


The tasks I am working on is:
* [x] my own task or dataset: 

## To reproduce

Steps to reproduce the behavior:

```
from transformers import ReformerForSequenceClassification, ReformerTokenizerFast
test = ReformerForSequenceClassification.from_pretrained('google/reformer-crime-and-punishment')
tokenizer = ReformerTokenizerFast.from_pretrained('google/reformer-crime-and-punishment')

input = tokenizer("this is a test", return_tensors='pt')
out = test(**input, labels = torch.zeros((1,1), dtype=torch.long))

out.loss.backward()
```

Error message:
```
TypeError                                 Traceback (most recent call last)
<ipython-input-155-db0c2d6dca2a> in <module>()
      6 out = test(**input, labels = torch.zeros((1,1), dtype=torch.long))
      7 
----> 8 out.loss.backward()


5 frames
/usr/local/lib/python3.7/dist-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    219                 retain_graph=retain_graph,
    220                 create_graph=create_graph)
--> 221         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    222 
    223     def register_hook(self, hook):

/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    130     Variable._execution_engine.run_backward(
    131         tensors, grad_tensors_, retain_graph, create_graph,
--> 132         allow_unreachable=True)  # allow_unreachable flag
    133 
    134 

/usr/local/lib/python3.7/dist-packages/torch/autograd/function.py in apply(self, *args)
     87     def apply(self, *args):
     88         # _forward_cls is defined by derived class
---> 89         return self._forward_cls.backward(self, *args)  # type: ignore
     90 
     91 

/usr/local/lib/python3.7/dist-packages/transformers/models/reformer/modeling_reformer.py in backward(***failed resolving arguments***)
   1673                 head_mask=head_mask[len(layers) - idx - 1],
   1674                 attention_mask=attention_mask,
-> 1675                 buckets=buckets,
   1676             )
   1677 

/usr/local/lib/python3.7/dist-packages/transformers/models/reformer/modeling_reformer.py in backward_pass(self, next_attn_output, hidden_states, grad_attn_output, grad_hidden_states, attention_mask, head_mask, buckets)
   1527 
   1528             # set seed to have correct dropout
-> 1529             torch.manual_seed(self.feed_forward_seed)
   1530             # g(Y_1)
   1531             res_hidden_states = self.feed_forward(next_attn_output)

/usr/local/lib/python3.7/dist-packages/torch/random.py in manual_seed(seed)
     30             `0xffff_ffff_ffff_ffff + seed`.
     31     """
---> 32     seed = int(seed)
     33     import torch.cuda
     34 

TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
```

## Expected behavior

No error.

## Workaround
Calling the seed init functions fixes the issue:
```
from transformers import ReformerForSequenceClassification, ReformerTokenizerFast
test = ReformerForSequenceClassification.from_pretrained('google/reformer-crime-and-punishment')
tokenizer = ReformerTokenizerFast.from_pretrained('google/reformer-crime-and-punishment')

for l in [m for m in test.modules()][0].reformer.encoder.layers:
  l._init_feed_forward_seed()
  l._init_attention_seed()

input = tokenizer("this is a test", return_tensors='pt')
out = test(**input, labels = torch.zeros((1,1), dtype=torch.long))

out.loss.backward()
```

Also note that this doesn't occur when I use a custom config.


Comments:
duplicate of https://github.com/huggingface/transformers/issues/10370 more or less. Just need to do the same fixes as in the answer of #10370