BatchEncoding.to() throwing torch NameError in 4.2.0; identical code works in 4.1.1
Author: KhoomeiK
Pull request: no

## Environment info

- `transformers` version: 4.2.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.4.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Haven't explicitly set up any parallelization other than GPU acceleration and not sure it's relevant since this is an error in the tokenizer

This is on Google Colab with a GPU by the way.

### Who can help

@mfuntowicz (tokenizers)
@sgugger (recent commits to the relevant file)

## Information

Model I am using (Bert, XLNet ...): ALBERT (but problem seems to be in tokenizer)

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

See script in reproduce section.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

Sequence classification, but the problem arises when transporting a BatchEncoding object to a certain torch device.

## To reproduce

Steps to reproduce the behavior:

Run [this colab notebook](https://colab.research.google.com/drive/1Lpu8wE8-1SKGuVLpRhK8VIy4dOWvKteF?usp=sharing). Alternatively...

1. Create a colab instance with GPU acceleration
2. Install torch, sentencepiece, transformers==4.2.0
3. Run the code below

```python
import torch
from transformers import AlbertTokenizer

tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
tokens = tokenizer('hello world', return_tensors='pt')
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
tokens = tokens.to(device)
```

There is no output when running 4.1.1 (expected) but the output when running 4.2.0 is below:

```
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-2-ad769dc72ebd> in <module>()
      6 tokens = tokenizer('hello world', return_tensors='pt')
      7 device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
----> 8 tokens = tokens.to(device)

1 frames
/usr/local/lib/python3.6/dist-packages/transformers/file_utils.py in wrapper(*args, **kwargs)
   1302     def wrapper(*args, **kwargs):
   1303         if is_torch_available():
-> 1304             return func(*args, **kwargs)
   1305         else:
   1306             raise ImportError(f"Method `{func.__name__}` requires PyTorch.")

/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py in to(self, device)
    802         # Otherwise it passes the casts down and casts the LongTensor containing the token idxs
    803         # into a HalfTensor
--> 804         if isinstance(device, str) or isinstance(device, torch.device) or isinstance(device, int):
    805             self.data = {k: v.to(device=device) for k, v in self.data.items()}
    806         else:

NameError: name 'torch' is not defined
```

## Expected behavior

There should be no console output and the tokens should be transferred to the correct device. The code works perfectly fine in version 4.1.1 of `transformers`.

I'll roll back to 4.1.1 for now, looking forward to any updates. Thanks!

Comments:
Hi, thanks for raising an issue!
Indeed, this is problematic. We're going to do a patch release this morning (v4.2.1) with a fix for this.

Ths fix is here: https://github.com/huggingface/transformers/pull/9584

It should be merged in a couple of hours, after which we'll release a patch.