Repository: huggingface/tokenizers (language: rust)

Issue:
Python3.12 build for Windows is not available
On Windows platforms using Python3.12 x64. The following installation fails:

```sh
$ pip install tokenizers
Collecting tokenizers
  Downloading tokenizers-0.15.0.tar.gz (318 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 318.5/318.5 kB 858.8 kB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]

      Cargo, the Rust package manager, is not installed or is not on PATH.
      This package requires Rust and Cargo to compile extensions. Install it through
      the system's package manager or via https://rustup.rs/

      Checking for Rust toolchain....
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
```
unable to install on python 3.12 via pip
Reproducing:

1. conda create -n py12 python=3.12
2. conda activate py12
3. pip install tokenizers
```
pip install tokenizers
Collecting tokenizers
  Using cached tokenizers-0.15.0.tar.gz (318 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]

      Cargo, the Rust package manager, is not installed or is not on PATH.
      This package requires Rust and Cargo to compile extensions. Install it through
      the system's package manager or via https://rustup.rs/

      Checking for Rust toolchain....
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.


```


Discussion / hints:
Sorry a release is long due for python 3.12, will get to it tomorrow hopefully 
You need to have cargo installed because we did not push wheels for mac with py>3.10. @Narsil should we try go up to 3.11 / 3.12 or are we limited by the runners ? 

> You need to have cargo installed because we did not push wheels for mac with py>3.10. @Narsil should we try go up to 3.11 / 3.12 or are we limited by the runners ?

No, I using x86 windows os, not mac. Additionally, I still cannot install tokenizers after `pip install cargo`. (In python 3.11, everything is fine even without cargo)
Sorry for the confusion. You have to [install rust](https://doc.rust-lang.org/book/ch01-01-installation.html#installing-rustup-on-windows), and probably not through pip. We compiled the wheels for python up to 3.11 not above
@ArthurZucker Would you be up to do a patch release with adding 3.12 support ?
Sure, I'll work on this this week 😉 
Hey @ArthurZucker would the lack of wheel compilation also be the reason for the following error?: `/lib/python3.11/site-packages/tokenizers/tokenizers.cpython-311-x86_64-linux-gnu.so: ELF load command past end of file` I'm on python 3.11.4.

I can't get chromadb to import tokenizers, it says it doesnt exist even though it does. I'm not sure if I should start a separate issue.

> ELF load command past end of file

This means corrupted file, probably unfinished download or something along those lines, delete the folder a install again I think.
See #1406 that will try to release for windows x python 3.12
Christmas delays planning a release for tomorrow 🤗 
Another option is to install tokenizers through conda:
`conda install conda-forge::tokenizers`

Fix patch:
diff --git a/.github/workflows/python-release.yml b/.github/workflows/python-release.yml
index e9d90d552..e6f8bfb97 100644
--- a/.github/workflows/python-release.yml
+++ b/.github/workflows/python-release.yml
@@ -42,19 +42,19 @@ jobs:
             ls: dir
             target: x86_64
             python-architecture: x64
-            interpreter: 3.7 3.8 3.9 3.10 3.11
+            interpreter: 3.7 3.8 3.9 3.10 3.11 3.12
           - os: windows
             ls: dir
             target: i686
             python-architecture: x86
-            interpreter: 3.7 3.8 3.9 3.10 3.11
+            interpreter: 3.7 3.8 3.9 3.10 3.11 3.12
               # - os: windows
               #   ls: dir
               #   target: aarch64
               #   interpreter: 3.11 3.12
           - os: macos
             target: aarch64
-            interpreter: 3.7 3.8 3.9 3.10 3.11 3.12 pypy3.8 pypy3.9 pypy3.10
+            interpreter: 3.7 3.8 3.9 3.10 3.11 3.12 3.13 pypy3.8 pypy3.9 pypy3.10
           - os: ubuntu
             platform: linux
             target: i686
@@ -65,7 +65,7 @@ jobs:
           - os: ubuntu
             platform: linux
             target: armv7
-            interpreter: 3.7 3.8 3.9 3.10 3.11 3.12
+            interpreter: 3.7 3.8 3.9 3.10 3.11 3.12 3.13
           # musllinux
           - os: ubuntu
             platform: linux
@@ -78,11 +78,11 @@ jobs:
           - os: ubuntu
             platform: linux
             target: ppc64le
-            interpreter: 3.7 3.8 3.9 3.10 3.11 3.12
+            interpreter: 3.7 3.8 3.9 3.10 3.11 3.12 3.13
           - os: ubuntu
             platform: linux
             target: s390x
-            interpreter: 3.7 3.8 3.9 3.10 3.11 3.12
+            interpreter: 3.7 3.8 3.9 3.10 3.11 3.12 3.13
         exclude:
           - os: windows
             target: aarch64
@@ -102,7 +102,7 @@ jobs:
       - name: set up python
         uses: actions/setup-python@v4
         with:
-          python-version: "3.11"
+          python-version: "3.12"
           architecture: ${{ matrix.python-architecture || 'x64' }}
 
       - run: pip install -U twine
@@ -114,7 +114,7 @@ jobs:
           working-directory: ./bindings/python
           manylinux: ${{ matrix.manylinux || 'auto' }}
           container: ${{ matrix.container }}
-          args: --release --out dist --interpreter ${{ matrix.interpreter || '3.7 3.8 3.9 3.10 3.11 3.12 pypy3.7 pypy3.8 pypy3.9 pypy3.10' }} ${{ matrix.extra-build-args }}
+          args: --release --out dist --interpreter ${{ matrix.interpreter || '3.7 3.8 3.9 3.10 3.11 3.12 3.13 pypy3.7 pypy3.8 pypy3.9 pypy3.10' }} ${{ matrix.extra-build-args }}
           rust-toolchain: stable
           docker-options: -e CI
 
@@ -165,7 +165,7 @@ jobs:
       - name: Install Python
         uses: actions/setup-python@v4
         with:
-          python-version: "3.10"
+          python-version: "3.12"
           architecture: x64
 
       - name: Retrieve all wheels
@@ -178,4 +178,4 @@ jobs:
         working-directory: ./bindings/python
         run: |
           pip install twine
-          twine upload dist/* -u __token__ -p "$PYPI_TOKEN"
+          twine upload dist/* -u __token__ -p "$PYPI_TOKEN"
\ No newline at end of file


Test patch: