Guard imports of PreTrainedTokenizerFast on is_tokenizers_available
Author: hvaara
Pull request: yes

# What does this PR do?

This PR guards the imports of `PreTrainedTokenizerFast` which depends on [huggingface/tokenizers](https://github.com/huggingface/tokenizers). This class could be imported when the `tokenizers` library isn't installed in which case an exception is raised.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->




## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer: @stas00, Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @hvaara, thanks for creating this PR!

Am I correct in saying this PR is to address an error when running: `from transformers import AutoTokenizer` if `tokenizers` isn't installed? 

I can see there's a similar import of `PreTrainedTokenizerFast` in `pipelines/__init__.py`. Could you a safe import there as well? It seems it's just for types so can probably be placed in the `if TYPE_CHECKING` logic. 

Hi @amyeroberts!

> Am I correct in saying this PR is to address an error when running: from transformers import AutoTokenizer if tokenizers isn't installed?

Yes, it will attempt to import `tokenizers` when it is not installed and raise an error.

> I can see there's a similar import of PreTrainedTokenizerFast in `pipelines/__init__.py`. Could you a safe import there as well?

SG! I created this PR in WIP mode mainly to see if the tests would pass. Handling this in `pipelines/__init__.py` was on my TODO list assuming this PR passed tests.

> It seems it's just for types so can probably be placed in the if TYPE_CHECKING logic.

Good feedback. I'll look into updating the logic to reflect this and also handle the case in `pipelines/__init__.py`.

Perhaps I should also create an additional (set of) test(s) to verify the test suite can be run when certain dependencies (like `tokenizers`) does not exist? Are there similar tests like this already? If so, can you please point me to them, and if not, how do you propose I create a test like that?

@hvaara Great! Looking forward to having this added and safe imports.

For the test questions, I'll hand this over to our expert @ydshieh who will know :) 

Hi @hvaara 

Thank you for the PR 🚀 

> Perhaps I should also create an additional (set of) test(s) to verify the test suite can be run when certain dependencies (like tokenizers) does not exist? Are there similar tests like this already? If so, can you please point me to them, and if not, how do you propose I create a test like that?

Regarding the testing,  did you see the test suite failed to collect or run some tests when `tokenizers` is not installed? (rather than being skipped).


Hi @hvaara Regarding 

> Perhaps I should also create an additional (set of) test(s) to verify the test suite can be run when certain dependencies (like tokenizers) does not exist? Are there similar tests like this already? If so, can you please point me to them, and if not, how do you propose I create a test like that?

I think we can keep the PR simple as it is.

On our CI, we make sure the required dependencies are installed. And if we see some errors caused by this issue, we will fix by installing the dependencies.

For community contributors, it's still much better for them to install the dependencies if they want/need to run the tests locally. Otherwise, the test results may be all green (i.e. pass), but (a lot) of the test methods are actually skipped. This may lead to a gap in the communication.

Thank you for the PR again!.

> > Perhaps I should also create an additional (set of) test(s) to verify the test suite can be run when certain dependencies (like tokenizers) does not exist? Are there similar tests like this already? If so, can you please point me to them, and if not, how do you propose I create a test like that?
> 
> I think we can keep the PR simple as it is.

SGTM.

I'll update the commit one more time, but that should be the last one assuming the tests pass and it LGTY. Sorry for the spam.

Thanks a lot for the help, and for merging the PR!