AttributeError with T5Tokenizer
Author: snat1505027
Pull request: no

I am trying to use **T5Tokenizer** and **t5-base** model to fine-tune on **SQuAD** dataset. But each time, when I run the tokenizer code I get errors (e.g, `'NoneType' object has no attribute 'encode'/'batch_encode_plus'/'encode_plus'`).

Example code
```
tokenizer = T5Tokenizer.from_pretrained('t5-base')
ids_neg = tokenizer.encode('negative 
')
ids_pos = tokenizer.encode('positive 
')
```
I get the following error:
> AttributeError                            Traceback (most recent call last)
> <ipython-input-19-f34cd55ac673> in <module>()
> ----> 1 ids_neg = tokenizer.encode('negative 
')
>       2 ids_pos = tokenizer.encode('positive 
')
>       3 len(ids_neg), len(ids_pos)
> 
> AttributeError: 'NoneType' object has no attribute 'encode'

Comments:
I think the errors could be more explicit here, here I think it comes from the fact that you don't have SentencePiece installed. Can you try to install it and let me know if it fixes your issue?

Hi @LysandreJik. I had the same issue, with sentencepiece installed. I also notice that my previous notebooks with T5Tokenizer and t5-base don't also work as well.

Here's my error:

```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-42-63aa129d4b0c> in <module>()
----> 1 dataset = ParaphraseDataset(tokenizer, 'data', 'dev', 256)
      2 print("Val dataset: ",len(dataset))
      3 
      4 data = dataset[61]
      5 print(tokenizer.decode(data['source_ids']))

1 frames
<ipython-input-39-2fa4af2cad5a> in __init__(self, tokenizer, data_dir, type_path, max_len)
     12         self.targets = []
     13 
---> 14         self._build()
     15 
     16     def __len__(self):

<ipython-input-39-2fa4af2cad5a> in _build(self)
     34 
     35             # tokenize inputs
---> 36             tokenized_inputs = self.tokenizer.batch_encode_plus(
     37                 [input_], max_length=self.max_len, pad_to_max_length=True, return_tensors="pt", truncation='longest_first'
     38             )

AttributeError: 'NoneType' object has no attribute 'batch_encode_plus'
```

It seems to me that T5Tokenizer isn't loading the T5-base tokenizer properly

Hmmm I'm pretty sure this only happens when you don't have sentencepiece installed. Do you mind pasting your environment info as well as `pip list`? If you're running on colab, can you make sure you restart the runtime before trying again? Thank you for your patience.

Here's mine (sorry for the long scroll!) (PS: I just found that downgrading the version of the transformers package solves the issue):

```
Package                       Version        
----------------------------- ---------------
absl-py                       0.10.0         
aiohttp                       3.7.3          
alabaster                     0.7.12         
albumentations                0.1.12         
altair                        4.1.0          
appdirs                       1.4.4          
argon2-cffi                   20.1.0         
asgiref                       3.3.1          
astor                         0.8.1          
astropy                       4.1            
astunparse                    1.6.3          
async-generator               1.10           
async-timeout                 3.0.1          
atari-py                      0.2.6          
atomicwrites                  1.4.0          
attrs                         20.3.0         
audioread                     2.1.9          
autograd                      1.3            
Babel                         2.9.0          
backcall                      0.2.0          
beautifulsoup4                4.6.3          
bleach                        3.2.2          
blis                          0.4.1          
bokeh                         2.1.1          
Bottleneck                    1.3.2          
branca                        0.4.2          
bs4                           0.0.1          
CacheControl                  0.12.6         
cachetools                    4.2.1          
catalogue                     1.0.0          
certifi                       2020.12.5      
cffi                          1.14.4         
chainer                       7.4.0          
chardet                       3.0.4          
click                         7.1.2          
cloudpickle                   1.3.0          
cmake                         3.12.0         
cmdstanpy                     0.9.5          
colorlover                    0.3.0          
community                     1.0.0b1        
contextlib2                   0.5.5          
convertdate                   2.2.0          
coverage                      3.7.1          
coveralls                     0.5            
crcmod                        1.7            
cufflinks                     0.17.3         
cupy-cuda101                  7.4.0          
cvxopt                        1.2.5          
cvxpy                         1.0.31         
cycler                        0.10.0         
cymem                         2.0.5          
Cython                        0.29.21        
daft                          0.0.4          
dask                          2.12.0         
dataclasses                   0.8            
datascience                   0.10.6         
debugpy                       1.0.0          
decorator                     4.4.2          
defusedxml                    0.6.0          
descartes                     1.1.0          
dill                          0.3.3          
distributed                   1.25.3         
Django                        3.1.5          
dlib                          19.18.0        
dm-tree                       0.1.5          
docopt                        0.6.2          
docutils                      0.16           
dopamine-rl                   1.0.5          
earthengine-api               0.1.238        
easydict                      1.9            
ecos                          2.0.7.post1    
editdistance                  0.5.3          
en-core-web-sm                2.2.5          
entrypoints                   0.3            
ephem                         3.7.7.1        
et-xmlfile                    1.0.1          
fa2                           0.3.5          
fancyimpute                   0.4.3          
fastai                        1.0.61         
fastdtw                       0.3.4          
fastprogress                  1.0.0          
fastrlock                     0.5            
fbprophet                     0.7.1          
feather-format                0.4.1          
filelock                      3.0.12         
firebase-admin                4.4.0          
fix-yahoo-finance             0.0.22         
Flask                         1.1.2          
flatbuffers                   1.12           
folium                        0.8.3          
fsspec                        0.8.5          
future                        0.18.2         
gast                          0.3.3          
GDAL                          2.2.2          
gdown                         3.6.4          
gensim                        3.6.0          
geographiclib                 1.50           
geopy                         1.17.0         
gin-config                    0.4.0          
glob2                         0.7            
google                        2.0.3          
google-api-core               1.16.0         
google-api-python-client      1.7.12         
google-auth                   1.17.2         
google-auth-httplib2          0.0.4          
google-auth-oauthlib          0.4.2          
google-cloud-bigquery         1.21.0         
google-cloud-bigquery-storage 1.1.0          
google-cloud-core             1.0.3          
google-cloud-datastore        1.8.0          
google-cloud-firestore        1.7.0          
google-cloud-language         1.2.0          
google-cloud-storage          1.18.1         
google-cloud-translate        1.5.0          
google-colab                  1.0.0          
google-pasta                  0.2.0          
google-resumable-media        0.4.1          
googleapis-common-protos      1.52.0         
googledrivedownloader         0.4            
graphviz                      0.10.1         
grpcio                        1.32.0         
gspread                       3.0.1          
gspread-dataframe             3.0.8          
gym                           0.17.3         
h5py                          2.10.0         
HeapDict                      1.0.1          
holidays                      0.10.4         
holoviews                     1.13.5         
html5lib                      1.0.1          
httpimport                    0.5.18         
httplib2                      0.17.4         
httplib2shim                  0.0.3          
humanize                      0.5.1          
hyperopt                      0.1.2          
ideep4py                      2.0.0.post3    
idna                          2.10           
idna-ssl                      1.1.0          
image                         1.5.33         
imageio                       2.4.1          
imagesize                     1.2.0          
imbalanced-learn              0.4.3          
imblearn                      0.0            
imgaug                        0.2.9          
importlib-metadata            3.4.0          
importlib-resources           5.1.0          
imutils                       0.5.4          
inflect                       2.1.0          
iniconfig                     1.1.1          
intel-openmp                  2021.1.2       
intervaltree                  2.1.0          
ipykernel                     4.10.1         
ipython                       5.5.0          
ipython-genutils              0.2.0          
ipython-sql                   0.3.9          
ipywidgets                    7.6.3          
itsdangerous                  1.1.0          
jax                           0.2.7          
jaxlib                        0.1.57+cuda101 
jdcal                         1.4.1          
jedi                          0.18.0         
jieba                         0.42.1         
Jinja2                        2.11.2         
joblib                        1.0.0          
jpeg4py                       0.1.4          
jsonschema                    2.6.0          
jupyter                       1.0.0          
jupyter-client                5.3.5          
jupyter-console               5.2.0          
jupyter-core                  4.7.0          
jupyterlab-pygments           0.1.2          
jupyterlab-widgets            1.0.0          
kaggle                        1.5.10         
kapre                         0.1.3.1        
Keras                         2.4.3          
Keras-Preprocessing           1.1.2          
keras-vis                     0.4.1          
kiwisolver                    1.3.1          
knnimpute                     0.1.0          
korean-lunar-calendar         0.2.1          
librosa                       0.8.0          
lightgbm                      2.2.3          
llvmlite                      0.34.0         
lmdb                          0.99           
lucid                         0.3.8          
LunarCalendar                 0.0.9          
lxml                          4.2.6          
Markdown                      3.3.3          
MarkupSafe                    1.1.1          
matplotlib                    3.2.2          
matplotlib-venn               0.11.6         
missingno                     0.4.2          
mistune                       0.8.4          
mizani                        0.6.0          
mkl                           2019.0         
mlxtend                       0.14.0         
more-itertools                8.6.0          
moviepy                       0.2.3.5        
mpmath                        1.1.0          
msgpack                       1.0.2          
multidict                     5.1.0          
multiprocess                  0.70.11.1      
multitasking                  0.0.9          
murmurhash                    1.0.5          
music21                       5.5.0          
natsort                       5.5.0          
nbclient                      0.5.1          
nbconvert                     5.6.1          
nbformat                      5.1.2          
nest-asyncio                  1.4.3          
networkx                      2.5            
nibabel                       3.0.2          
nltk                          3.2.5          
notebook                      5.3.1          
np-utils                      0.5.12.1       
numba                         0.51.2         
numexpr                       2.7.2          
numpy                         1.19.5         
nvidia-ml-py3                 7.352.0        
oauth2client                  4.1.3          
oauthlib                      3.1.0          
okgrade                       0.4.3          
opencv-contrib-python         4.1.2.30       
opencv-python                 4.1.2.30       
openpyxl                      2.5.9          
opt-einsum                    3.3.0          
osqp                          0.6.2.post0    
packaging                     20.8           
palettable                    3.3.0          
pandas                        1.1.5          
pandas-datareader             0.9.0          
pandas-gbq                    0.13.3         
pandas-profiling              1.4.1          
pandocfilters                 1.4.3          
panel                         0.9.7          
param                         1.10.1         
parso                         0.8.1          
pathlib                       1.0.1          
patsy                         0.5.1          
pexpect                       4.8.0          
pickleshare                   0.7.5          
Pillow                        7.0.0          
pip                           19.3.1         
pip-tools                     4.5.1          
plac                          1.1.3          
plotly                        4.4.1          
plotnine                      0.6.0          
pluggy                        0.7.1          
pooch                         1.3.0          
portpicker                    1.3.1          
prefetch-generator            1.0.1          
preshed                       3.0.5          
prettytable                   2.0.0          
progressbar2                  3.38.0         
prometheus-client             0.9.0          
promise                       2.3            
prompt-toolkit                1.0.18         
protobuf                      3.12.4         
psutil                        5.4.8          
psycopg2                      2.7.6.1        
ptyprocess                    0.7.0          
py                            1.10.0         
pyarrow                       0.14.1         
pyasn1                        0.4.8          
pyasn1-modules                0.2.8          
pycocotools                   2.0.2          
pycparser                     2.20           
pyct                          0.4.8          
pydata-google-auth            1.1.0          
pydot                         1.3.0          
pydot-ng                      2.0.0          
pydotplus                     2.0.2          
PyDrive                       1.3.1          
pyemd                         0.5.1          
pyglet                        1.5.0          
Pygments                      2.6.1          
pygobject                     3.26.1         
pymc3                         3.7            
PyMeeus                       0.3.7          
pymongo                       3.11.2         
pymystem3                     0.2.0          
pynndescent                   0.5.1          
PyOpenGL                      3.1.5          
pyparsing                     2.4.7          
pyrsistent                    0.17.3         
pysndfile                     1.3.8          
PySocks                       1.7.1          
pystan                        2.19.1.1       
pytest                        3.6.4          
python-apt                    1.6.5+ubuntu0.5
python-chess                  0.23.11        
python-dateutil               2.8.1          
python-louvain                0.15           
python-slugify                4.0.1          
python-utils                  2.5.3          
pytorch-lightning             1.1.6          
pytz                          2018.9         
pyviz-comms                   2.0.1          
PyWavelets                    1.1.1          
PyYAML                        5.3.1          
pyzmq                         21.0.1         
qdldl                         0.1.5.post0    
qtconsole                     5.0.2          
QtPy                          1.9.0          
regex                         2019.12.20     
requests                      2.23.0         
requests-oauthlib             1.3.0          
resampy                       0.2.2          
retrying                      1.3.3          
rpy2                          3.2.7          
rsa                           4.7            
sacremoses                    0.0.43         
scikit-image                  0.16.2         
scikit-learn                  0.22.2.post1   
scipy                         1.4.1          
screen-resolution-extra       0.0.0          
scs                           2.1.2          
seaborn                       0.11.1         
Send2Trash                    1.5.0          
sentencepiece                 0.1.95         
setuptools                    51.3.3         
setuptools-git                1.2            
Shapely                       1.7.1          
simplegeneric                 0.8.1          
six                           1.15.0         
sklearn                       0.0            
sklearn-pandas                1.8.0          
smart-open                    4.1.2          
snowballstemmer               2.1.0          
sortedcontainers              2.3.0          
SoundFile                     0.10.3.post1   
spacy                         2.2.4          
Sphinx                        1.8.5          
sphinxcontrib-serializinghtml 1.1.4          
sphinxcontrib-websupport      1.2.4          
SQLAlchemy                    1.3.22         
sqlparse                      0.4.1          
srsly                         1.0.5          
statsmodels                   0.10.2         
sympy                         1.1.1          
tables                        3.4.4          
tabulate                      0.8.7          
tblib                         1.7.0          
tensorboard                   2.4.1          
tensorboard-plugin-wit        1.8.0          
tensorboardcolab              0.0.22         
tensorflow                    2.4.1          
tensorflow-addons             0.8.3          
tensorflow-datasets           4.0.1          
tensorflow-estimator          2.4.0          
tensorflow-gcs-config         2.4.0          
tensorflow-hub                0.11.0         
tensorflow-metadata           0.27.0         
tensorflow-privacy            0.2.2          
tensorflow-probability        0.12.1         
termcolor                     1.1.0          
terminado                     0.9.2          
testpath                      0.4.4          
text-unidecode                1.3            
textblob                      0.15.3         
textgenrnn                    1.4.1          
Theano                        1.0.5          
thinc                         7.4.0          
tifffile                      2020.9.3       
tokenizers                    0.8.1rc2       
toml                          0.10.2         
toolz                         0.11.1         
torch                         1.7.0+cu101    
torchsummary                  1.5.1          
torchtext                     0.3.1          
torchvision                   0.8.1+cu101    
tornado                       5.1.1          
tqdm                          4.41.1         
traitlets                     4.3.3          
transformers                  3.3.0          
tweepy                        3.6.0          
typeguard                     2.7.1          
typing-extensions             3.7.4.3        
tzlocal                       1.5.1          
umap-learn                    0.5.0          
uritemplate                   3.0.1          
urllib3                       1.24.3         
vega-datasets                 0.9.0          
wasabi                        0.8.1          
wcwidth                       0.2.5          
webencodings                  0.5.1          
Werkzeug                      1.0.1          
wheel                         0.36.2         
widgetsnbextension            3.5.1          
wordcloud                     1.5.0          
wrapt                         1.12.1         
xarray                        0.15.1         
xgboost                       0.90           
xkit                          0.0.0          
xlrd                          1.1.0          
xlwt                          1.3.0          
yarl                          1.6.3          
yellowbrick                   0.9.1          
zict                          2.0.0          
zipp                          3.4.0
```

Thank you for sharing! In previous packages, we needed `sentencepiece` so it was installed automatically. It's not anymore, with I think was the issue here. Will look into it further.

> I think the errors could be more explicit here, here I think it comes from the fact that you don't have SentencePiece installed. Can you try to install it and let me know if it fixes your issue?

Yes, I have already installed **SentencePiece**.

It eventually worked for me with the following re-installation:

```
!pip install transformers==2.9.0 
!pip install pytorch_lightning==0.7.5
```

Maybe the error was due to the specific version.

You need sentencepiece:  **_!pip install sentencepiece_**

However, if you are using colab notebook you have to **_restart the runtime for it to work._** after installing sentencepiece_

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.