CLIPTokenizer problems in from pretrained on version 4.25.1
Author: JJ-IsShort
Pull request: no

### System Info

WARNING:tensorflow:From D:\Stable Diffusion\SD New\stable-diffusion-webui\venv\lib\site-packages\transformers\commands\env.py:52: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2023-03-11 10:21:20.969146: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.25.1
- Platform: Windows-10-10.0.19044-SP0
- Python version: 3.10.7
- Huggingface_hub version: 0.12.0
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): 2.11.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in> Yes
- Using distributed or parallel set-up in script?: <fill in> I don't know

### Who can help?

@amyeroberts
@ArthurZucker and @younesbelkada

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

This is what I placed in the Python shell.
```pycon
Python 3.10.7 (tags/v3.10.7:6cc6b13, Sep  5 2022, 14:08:36) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import transformers
>>> from transformers import CLIPTokenizer
>>> tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-large-patch14")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\Stable Diffusion\SD New\stable-diffusion-webui\venv\lib\site-packages\transformers\tokenization_utils_base.py", line 1785, in from_pretrained
    raise EnvironmentError(
OSError: Can't load tokenizer for 'openai/clip-vit-large-patch14'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'openai/clip-vit-large-patch14' is the correct path to a directory containing all relevant files for a CLIPTokenizer tokenizer.
```

Other than that, I'm also getting the same issue in the same place just with a longer stack trace in the AUTOMATIC1111 webui and the kohya scripts. Also, in case this helps, "models--openai--clip-vit-large-patch14" does get created during the running of the scripts, but it is empty.

Edit: With the webui, when I set the version to 4.19.2 it works, except for the dreambooth extension. Didn't test with the kohya script and version 4.19.2.

### Expected behavior

I would expect that the Tokenizer would load properly and that the scripts I was running would work.

Comments:
Hey!
As you mention, you are creating an empty directory with the same name. The priority is given to local folders, which explains why you are having this issue. Either delete the empty folder or save a tokenizer inside it 😉 

I have tried to delete the empty folder, but it just keeps coming back, even when just running the code I mentioned before that I placed into the console. I also don't know how to put a tokenizer into this folder that can't be deleted. Could you please help with that?

To put the tokenizer in the folder run: 
```python 
tokenizer.save_pretrained("path_to_folder")
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I am so sorry that I haven't replied in so long. Work IRL took over my life. Also, transformers 4.19.2 worked well enough because I could just use colab for the khoya scripts.

The issue is that the tokenizer won't even be created in the first place. I again tried putting this in the python shell in the webui's virtual env:
```pycon
Python 3.10.7 (tags/v3.10.7:6cc6b13, Sep  5 2022, 14:08:36) [MSC v.1933 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import transformers
>>> from transformers import CLIPTokenizer
>>> tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-large-patch14")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\Stable Diffusion\SD 3\stable-diffusion-webui\venv\lib\site-packages\transformers\tokenization_utils_base.py", line 1785, in from_pretrained
    raise EnvironmentError(
OSError: Can't load tokenizer for 'openai/clip-vit-large-patch14'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'openai/clip-vit-large-patch14' is the correct path to a directory containing all relevant files for a CLIPTokenizer tokenizer.tokenizer.save_pretrained("path_to_folder")
```
tokenizer doesn't even get created, so I can't use ```tokenizer.save_pretrained("path_to_folder")``` on it. I also tried loading 'openai/clip-vit-base-patch16' and it got the same error. Is there another way to save the transformer snapshot into that folder?

That very strange, I can't reproduce your error at all. If you open any colab, the script that you share will work. 
Quick fixes are probably: `pip install --upgrade transformers`. 

I set the transformers version to 4.28.1 and still got the issue. I also tried version 4.26.0. The latest version, 4.28.1, did not work. Something that I thought was weird was where the files were downloaded to. With version 4.19.2, the files download fine. There are a lot of weirdly named files in ```.cache\huggingface\transformers``` one file with a seemingly random extension and random name, followed by a file with the same name, including the extension, plus ```.json```. They all have really small file sizes. ```.cache\clip``` has the ```ViT-L-14.pt``` file in it. It has the file size I would expect. In versions 4.28.1, 4.26.0, and 4.25.1, the file tries to save in ```.cache\huggingface\hub\models--openai--clip-vit-large-patch14```.  The docs say that it should be saved in the transformers folder in the huggingface directory. This is what I found weird. The docs didn't match up with what I was seeing. I also couldn't change the ```.cache``` directory using the environment variable ```HF_HOME```. Maybe I'm using the wrong hf version?

Edit: I actually think I didn't press apply, so maybe that's why the cache location didn't change.