Undefined variable in: scripts/check_tokenizers.py
Author: vignesh1507
Pull request: no
Labels: name: Core: Tokenization

name: bug

### System Info

Python 3.12.4

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`if check_diff(spm_ids[first : first + i], tok_ids[first : first + j], sp, tok) and check_details(
                        line,
                        spm_ids[first + i : last],
                        tok_ids[first + j : last],
                        slow,
                        fast,`

### Expected behavior

Undefined Variables: sp and tok are not defined anywhere within the check_details function or its enclosing scopes. This will result in a NameError when the code attempts to execute this line.

Comments:
Hey @vignesh1507, would you like to open a PR to fix this?

I opened a draft PR for this issue #33702 with what I believe are the correct variables to pass in this case, but I need to verify this further since I couldn't run it in my environment. I am getting the error:
```
AttributeError: module transformers has no attribute Phi3Tokenizer
```
Which makes me wonder whether this is due to the script being unmaintained. Could also just be my environment not being set up correctly though.

I also have had trouble understanding the purpose of the script, there is little documentation available and couldn't find much online. I would be happy to also document better the existence of this script with a bit of guidance. Do you think this could warrant an issue, @LysandreJik?

Yeah i can assist you with the documentation part and can you tell me more about the error you are facing on your local pc?

@vignesh1507 Sure, this is the complete traceback I am getting:

```
Traceback (most recent call last):
  File "/home/nicodea/hf/transformers/scripts/check_tokenizers.py", line 12, in <module>
    TOKENIZER_CLASSES = {
  File "/home/nicodea/hf/transformers/scripts/check_tokenizers.py", line 13, in <dictcomp>
    name: (getattr(transformers, name), getattr(transformers, name + "Fast")) for name in SLOW_TO_FAST_CONVERTERS
  File "/home/nicodea/hf/transformers/src/transformers/utils/import_utils.py", line 1757, in __getattr__
    raise AttributeError(f"module {self.__name__} has no attribute {name}")
AttributeError: module transformers has no attribute Phi3Tokenizer
```

Details about my environment:
* WSL 2 on Windows 11
* Ubuntu 22.04.2 LTS
* Conda virtual environment
* Python 3.10.14
* Ran `pip install -e ".[dev]"` at commit 68049b17a

~EDIT: upon reinstalling the environment (at commit f3c1a172b) with `pip install -e .[quality]` as suggested [here](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request) I am now no longer getting that error. Now, upon running `pip install -e ".[sentencepiece]"`, I am getting:~
```
<redacted>
```

Thanks for the response @niqodea I'll look into the error and get back to you with a possible solution.

Hey @vignesh1507, I updated the comment, never mind about the second error - I was using a very old version of `transformers`  because I pulled from my very outdated `origin/main`, my bad. The first error is the one I am still experiencing, and I can't seem to have it fixed even after installing with `.[quality]`.

you can try upgrading the python version to 3.12.4, maybe it would work and till then i'll find out why you're getting the first error which you have mentioned earlier. Again thanks for the response and i'll get back to you with a possible solution for your error. 

Peace ✌️ 

Hey @vignesh1507, I tried upgrading to Python 3.12.4 and now it's complaining about versions when I run `pip install -e .[dev]`. 

```
Obtaining file:///home/nicodea/hf/transformers
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Collecting filelock (from transformers==4.46.0.dev0)
  Using cached filelock-3.16.1-py3-none-any.whl.metadata (2.9 kB)
Collecting huggingface-hub<1.0,>=0.23.2 (from transformers==4.46.0.dev0)
  Using cached huggingface_hub-0.25.1-py3-none-any.whl.metadata (13 kB)
Collecting numpy>=1.17 (from transformers==4.46.0.dev0)
  Downloading numpy-2.1.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
Collecting packaging>=20.0 (from transformers==4.46.0.dev0)
  Using cached packaging-24.1-py3-none-any.whl.metadata (3.2 kB)
Collecting pyyaml>=5.1 (from transformers==4.46.0.dev0)
  Downloading PyYAML-6.0.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
Collecting regex!=2019.12.17 (from transformers==4.46.0.dev0)
  Downloading regex-2024.9.11-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (40 kB)
Collecting requests (from transformers==4.46.0.dev0)
  Using cached requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
Collecting tokenizers<0.21,>=0.20 (from transformers==4.46.0.dev0)
  Downloading tokenizers-0.20.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)
Collecting safetensors>=0.4.1 (from transformers==4.46.0.dev0)
  Downloading safetensors-0.4.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB)
Collecting tqdm>=4.27 (from transformers==4.46.0.dev0)
  Using cached tqdm-4.66.5-py3-none-any.whl.metadata (57 kB)
INFO: pip is looking at multiple versions of transformers[dev] to determine which version is compatible with other requirements. This could take a while.
ERROR: Ignored the following versions that require a different python version: 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python >=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python >=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11
ERROR: Could not find a version that satisfies the requirement tensorflow<2.16,>2.9; extra == "dev" (from transformers[dev]) (from versions: 2.16.0rc0, 2.16.1, 2.16.2, 2.17.0rc0, 2.17.0rc1, 2.17.0, 2.18.0rc0)
ERROR: No matching distribution found for tensorflow<2.16,>2.9; extra == "dev"
Exit code: 1
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.