Why does datasets get imported when running "from transformers.models.roberta.tokenization_roberta_fast import RobertaTokenizerFast"
Author: vgoklani
Pull request: no
Labels: name: wontfix

When running 

    from transformers.models.roberta.tokenization_roberta_fast import RobertaTokenizerFast

I get this warning:

    ImportWarning: To use `datasets`, the module `pyarrow>=0.16.0` is required, and the current version of `pyarrow` doesn't match this condition.  If you are running this in a Google Colab, you should probably just restart the runtime to use the right version of `pyarrow`.

Why is datasets getting imported, when we import the tokenizer? 

Thanks! :)

Comments:
hi @vgoklani 

`datasets` is not required for tokenizers, so it's unlikely to get this error when just importing the tokenizer. Are you running any examples scripts? because those require `datasets` lib

Hi @patil-suraj Happy New Year!

Here is the stack trace:

    root@b5d80f9670ea:~/src# ipython
    Python 3.8.5 (default, Sep  4 2020, 07:30:14)
    Type 'copyright', 'credits' or 'license' for more information
    IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.

    In [1]: from transformers.models.roberta.tokenization_roberta_fast import RobertaTokenizerFast
    2021-01-01 12:03:22.340215: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
    ---------------------------------------------------------------------------
    ImportWarning                             Traceback (most recent call last)
    <ipython-input-1-2758fed1e79c> in <module>
    ----> 1 from transformers.models.roberta.tokenization_roberta_fast import RobertaTokenizerFast

    /opt/conda/lib/python3.8/site-packages/transformers/__init__.py in <module>
         32     absl.logging._warn_preinit_stderr = False
         33
    ---> 34 from . import dependency_versions_check
         35
         36 # Configuration

    /opt/conda/lib/python3.8/site-packages/transformers/dependency_versions_check.py in <module>
         32         if pkg == "tokenizers":
         33             # must be loaded here, or else tqdm check may fail
    ---> 34             from .file_utils import is_tokenizers_available
         35
         36             if not is_tokenizers_available():

    /opt/conda/lib/python3.8/site-packages/transformers/file_utils.py in <module>
        101
        102 try:
    --> 103     import datasets  # noqa: F401
        104
        105     # Check we're not importing a "datasets" directory somewhere

    /opt/conda/lib/python3.8/site-packages/datasets/__init__.py in <module>
         51
         52 if int(pyarrow.__version__.split(".")[1]) < 16 and int(pyarrow.__version__.split(".")[0]) == 0:
    ---> 53     raise ImportWarning(
         54         "To use `datasets`, the module `pyarrow>=0.16.0` is required, and the current version of `pyarrow` doesn't match this condition.\n"
         55         "If you are running this in a Google Colab, you should probably just restart the runtime to use the right version of `pyarrow`."

    ImportWarning: To use `datasets`, the module `pyarrow>=0.16.0` is required, and the current version of `pyarrow` doesn't match this condition.
    If you are running this in a Google Colab, you should probably just restart the runtime to use the right version of `pyarrow`.


---

An older version of pyarrow was installed, but regardless, this happens immediately after the import. Upgrading pyarrow makes this warning disappear, but regardless, this shouldn't happen.


cc @sgugger 

This is because transformers imports all optional dependencies (like datasets) during its init. There will be some work to avoid doing that in the coming weeks.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.