Installation Error - Failed building wheel for tokenizers
Author: victorlongo
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

name: Installation

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): N/A

Language I am using the model on (English, Chinese ...): N/A

The problem arises when using:
* [X] the official example scripts: (give details below)

Problem arises in transformers installation on Microsoft Windows 10 Pro, version 10.0.17763 

After creating and activating the virtual environment, installing transformers is not possible, because the following error occurs:

"error: can not find Rust Compiler"
"ERROR: Failed building wheel for tokenizers"
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed d

The tasks I am working on is:
[X ] transformers installation

## To reproduce

Steps to reproduce the behavior:

1. From command line interface, create and activate a virtual environment by following the steps in this URL: https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/
2. Install transformers from source, by following the example in the topic From Source on this URL: https://github.com/huggingface/transformers

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```
-m pip --version
-m pip install --upgrade pip
-m pip install --user virtualenv
-m venv env
.\env\Scripts\activate
pip install transformers

ERROR: Command errored out with exit status 1:
   command: 'c:\users\vbrandao\env\scripts\python.exe' 'c:\users\vbrandao\env\lib\site-packages\pip\_vendor\pep517\_in_process.py' build_wheel 'C:\Users\vbrandao\AppData\Local\Temp\tmpj6evjmze'
       cwd: C:\Users\vbrandao\AppData\Local\Temp\pip-install-sza2_lmj\tokenizers
  Complete output (10 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib
  creating build\lib\tokenizers
  copying tokenizers\__init__.py -> build\lib\tokenizers
  running build_ext
  running build_rust
  error: Can not find Rust compiler
  ----------------------------------------
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed directly

```



## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

Installation of transformers should be complete.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: N/A - installation step
- Platform: Command Line Interface / Virtual Env
- Python version: python 3.8
- PyTorch version (GPU?): N/A
- Tensorflow version (GPU?): N/A
- Using GPU in script?: N/A
- Using distributed or parallel set-up in script?: N/A
![tokenizers_intallation_error](https://user-images.githubusercontent.com/17074908/74371705-06b3f680-4db8-11ea-8a2d-5f920cb3caab.PNG)


Comments:
Having the exact same issue on a Linux machine!

Environment: macOS Mojave Ver 10.14.6
Tried installing both from pip and source. Same issue:
> Successfully built transformers
> Failed to build tokenizers 

Result was that Transformers was not installed (not listed in pip freeze)

This however should work - seems like you just won't get the the new tokenizers:
pip install transformers==2.4.1

@GDBSD I had the same issue on the same OS version and also tried pip and source. Your version specification worked. 

Had the same issue on MacOS Mojave when doing pip3 install. Tried pip2 install, it worked but I got another error when running my script telling me I should really be using python 3.

I tried @GDBSD 's answer, but I got this error: 

```
ERROR: Exception:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/py_compile.py", line 143, in compile
    _optimize=optimize)
  File "<frozen importlib._bootstrap_external>", line 791, in source_to_code
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/private/var/folders/g0/5zwy4mtx7579v5x6rxqb083r0000gn/T/pip-unpacked-wheel-k410h9s0/sacremoses/sent_tokenize.py", line 69
    if re.search(IS_EOS, token)
                              ^
SyntaxError: invalid syntax

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/compileall.py", line 159, in compile_file
    invalidation_mode=invalidation_mode)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/py_compile.py", line 147, in compile
    raise py_exc
py_compile.PyCompileError:   File "/private/var/folders/g0/5zwy4mtx7579v5x6rxqb083r0000gn/T/pip-unpacked-wheel-k410h9s0/sacremoses/sent_tokenize.py", line 69
    if re.search(IS_EOS, token)
                              ^
SyntaxError: invalid syntax


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
    status = self.run(options, args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 404, in run
    use_user_site=options.use_user_site,
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip/_internal/req/__init__.py", line 71, in install_given_reqs
    **kwargs
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 815, in install
    warn_script_location=warn_script_location,
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip/_internal/operations/install/wheel.py", line 614, in install_wheel
    warn_script_location=warn_script_location,
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip/_internal/operations/install/wheel.py", line 338, in install_unpacked_wheel
    compileall.compile_dir(source, force=True, quiet=True)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/compileall.py", line 97, in compile_dir
    legacy, optimize, invalidation_mode):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/compileall.py", line 169, in compile_file
    msg = err.msg.encode(sys.stdout.encoding,
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip/_internal/utils/misc.py", line 554, in encoding
    return self.orig_stream.encoding
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/codecs.py", line 409, in __getattr__
    return getattr(self.stream, name)
AttributeError: '_io.BufferedWriter' object has no attribute 'encoding'
```

yes I had the same issue with `pip3.6 install`

Can you all run `python transformers-cli env` and post the output here? It provides some useful information about your platform that might be helpful to debug.

Hi, I had the same problem and resolved it by installing rust.
"error: Can not find Rust compiler"

For MacOS, I used "curl https://sh.rustup.rs -sSf | sh". I also found that it needed a nightly version of rust, so you have to specify that in the install options. 

Hi, I also had the same problem with my initial installation of the library. After some time, I realized that my anaconda version was on 32Bit. You can check your version with 
`python -c "import struct;print( 8 * struct.calcsize('P'))"`
The output should be 64.
If it is 32 then you have to reinstall your IDE


@Wild3d I can confirm after running your snippet that I am on a 64bit version 

@gardnerds after creating a new environment to try your solution that also worked for me. I didn't have rust installed before. It successfully built the wheel for tokenizers (PEP 517). 

@gardnerds also worked for me. Using python 3.7 and built from source using a clean conda env

Install Python 64-bit instead of 32-bit solved my same issue.

I was having the same issue on virtualenv over Mac OS Mojave. Managed to solve it and install Transformers 2.5.1 by manually install the last version of tokenizers (0.6.0) instead of 0.5.2 that is required in the transformer package.

pip install tokenizers

Git clone latest version of transformers:

git clone https://github.com/huggingface/transformers

Before running the installation edit  transformers/setup.py and change requirement of tokenizers to 0.6.0

 Line 93:   install_requires=[
                  "numpy",
                   "tokenizers == 0.6.0",

Then run as usual: 

cd transformers
pip install .

I assume that you could also skip the first step and just collect the package as you run the install. 
I'm quite new to this, so just wanted to share my take.

@dafraile That solves mine! Thank you very much!

@dafraile That helps, thanks a lot!

I managed to solve the issue by installing Rust compiler

- Install Rust [link](https://www.rust-lang.org/tools/install) `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`
- Restart the terminal
- `pip install transformers==2.5.1`

> Environment: macOS Mojave Ver 10.14.6
> Tried installing both from pip and source. Same issue:
> 
> > Successfully built transformers
> > Failed to build tokenizers
> 
> Result was that Transformers was not installed (not listed in pip freeze)
> 
> This however should work - seems like you just won't get the the new tokenizers:
> pip install transformers==2.4.1

This solution is working for me

> I managed to solve the issue by installing Rust compiler
> 
> * Install Rust [link](https://www.rust-lang.org/tools/install) `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`
> * Restart the terminal
> * `pip install transformers==2.5.1`

It works for me, thanks!
You can do `source $HOME/.cargo/env` instead of restarting the terminal.

@gardnerds, adding `$HOME/.cargo/bin` to PATH after installing rust fixed my installation. Thank you. 

@dafraile Thanks a lot. It solves my problem

@dafraile   Thanks! It works!

@AvivNavon Thanks ! Solved my problem too. (MacOS Mojave)
I install latest version of transformers though (2.8.0)
`pip install transformers` instead of `pip install transformers==2.5.1`

resolved this issue by installing Rust 

I resolved this issue by installing Rust - I initially did forget to restart the terminal first.
I'm using Mojave 10.14.5.
This thread is great! Btw I had no such issues on my Ubuntu 18.04 machine.

@phihung recommendation works. 

Just installing rust compiler works for me too (Thanks @phihung ) I'm on Mac Mojave 10.14.6. 
May be conda installation should be able to over come this? (don't know if pip can force install a 3rd party compiler)?

@dafraile Actually your solution is the closest one ! But now I saw that they just corrected that line in setup.py so it became tokenizers==0.7.0 now (and the newest tokenizers are 0.7.0).
So the real importance is that we should 
1. always update the transformers from the source 
2. (really important !) uninstall the old version before we reinstall the newest :p 


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I am facing a similar issue trying to build on a PowerPC with RedHat
I am getting errors when trying to build tokenizers:
```
Building wheels for collected packages: tokenizers
  Building wheel for tokenizers (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/aarbelle/.conda/envs/gbs/bin/python3.6 /home/aarbelle/.conda/envs/gbs/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpd6q9xccz
       cwd: /tmp/pip-install-ohxny31i/tokenizers
  Complete output (136 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib
  creating build/lib/tokenizers
  copying tokenizers/__init__.py -> build/lib/tokenizers
  creating build/lib/tokenizers/models
  copying tokenizers/models/__init__.py -> build/lib/tokenizers/models
  creating build/lib/tokenizers/decoders
  copying tokenizers/decoders/__init__.py -> build/lib/tokenizers/decoders
  creating build/lib/tokenizers/normalizers
  copying tokenizers/normalizers/__init__.py -> build/lib/tokenizers/normalizers
  creating build/lib/tokenizers/pre_tokenizers
  copying tokenizers/pre_tokenizers/__init__.py -> build/lib/tokenizers/pre_tokenizers
  creating build/lib/tokenizers/processors
  copying tokenizers/processors/__init__.py -> build/lib/tokenizers/processors
  creating build/lib/tokenizers/trainers
  copying tokenizers/trainers/__init__.py -> build/lib/tokenizers/trainers
  creating build/lib/tokenizers/implementations
  copying tokenizers/implementations/bert_wordpiece.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/__init__.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/byte_level_bpe.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/sentencepiece_bpe.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/base_tokenizer.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/char_level_bpe.py -> build/lib/tokenizers/implementations
  copying tokenizers/__init__.pyi -> build/lib/tokenizers
  copying tokenizers/models/__init__.pyi -> build/lib/tokenizers/models
  copying tokenizers/decoders/__init__.pyi -> build/lib/tokenizers/decoders
  copying tokenizers/normalizers/__init__.pyi -> build/lib/tokenizers/normalizers
  copying tokenizers/pre_tokenizers/__init__.pyi -> build/lib/tokenizers/pre_tokenizers
  copying tokenizers/processors/__init__.pyi -> build/lib/tokenizers/processors
  copying tokenizers/trainers/__init__.pyi -> build/lib/tokenizers/trainers
  running build_ext
  running build_rust
      Updating crates.io index
      Updating git repository `https://github.com/n1t0/rayon-cond`
  warning: unused manifest key: target.x86_64-apple-darwin.rustflags
     Compiling proc-macro2 v1.0.21
     Compiling unicode-xid v0.2.1
     Compiling autocfg v1.0.1
     Compiling syn v1.0.41
     Compiling libc v0.2.77
     Compiling lazy_static v1.4.0
     Compiling cfg-if v0.1.10
     Compiling memchr v2.3.3
     Compiling serde_derive v1.0.116
     Compiling scopeguard v1.1.0
     Compiling serde v1.0.116
     Compiling maybe-uninit v2.0.0
     Compiling regex-syntax v0.6.18
     Compiling ryu v1.0.5
     Compiling rayon-core v1.8.1
     Compiling getrandom v0.1.15
     Compiling serde_json v1.0.57
     Compiling smallvec v1.4.2
     Compiling itoa v0.4.6
     Compiling inventory v0.1.9
     Compiling pkg-config v0.3.18
     Compiling proc-macro-hack v0.5.18
     Compiling bitflags v1.2.1
     Compiling cc v1.0.60
     Compiling unicode-width v0.1.8
     Compiling either v1.6.1
       Running `rustc --crate-name build_script_build --edition=2018 /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-1.0.21/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=93385cb1e678e330 -C extra-filename=-93385cb1e678e330 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/build/proc-macro2-93385cb1e678e330 -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name unicode_xid /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-xid-0.2.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' -C metadata=cac161967aa527e1 -C extra-filename=-cac161967aa527e1 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name autocfg /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/autocfg-1.0.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=ddb9624730d1e52a -C extra-filename=-ddb9624730d1e52a --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build --edition=2018 /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/syn-1.0.41/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="clone-impls"' --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="extra-traits"' --cfg 'feature="full"' --cfg 'feature="parsing"' --cfg 'feature="printing"' --cfg 'feature="proc-macro"' --cfg 'feature="quote"' --cfg 'feature="visit"' -C metadata=9988fc7a157e69c9 -C extra-filename=-9988fc7a157e69c9 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/build/syn-9988fc7a157e69c9 -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.77/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=5a4798f2b06c36bd -C extra-filename=-5a4798f2b06c36bd --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/build/libc-5a4798f2b06c36bd -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name cfg_if --edition=2018 /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/cfg-if-0.1.10/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=a7dbefe7725970f6 -C extra-filename=-a7dbefe7725970f6 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name lazy_static /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/lazy_static-1.4.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=09f05f31cfc64306 -C extra-filename=-09f05f31cfc64306 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/memchr-2.3.3/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' --cfg 'feature="use_std"' -C metadata=a8f56f28f9bbd928 -C extra-filename=-a8f56f28f9bbd928 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/build/memchr-a8f56f28f9bbd928 -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_derive-1.0.116/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' -C metadata=d850080603f4774e -C extra-filename=-d850080603f4774e --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/build/serde_derive-d850080603f4774e -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name scopeguard /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/scopeguard-1.1.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=91afa33e60eb09b1 -C extra-filename=-91afa33e60eb09b1 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.116/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="serde_derive"' --cfg 'feature="std"' -C metadata=1a02cab7c16e427d -C extra-filename=-1a02cab7c16e427d --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/build/serde-1a02cab7c16e427d -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/maybe-uninit-2.0.0/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no -C metadata=9f94ee50e1295f1f -C extra-filename=-9f94ee50e1295f1f --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/build/maybe-uninit-9f94ee50e1295f1f -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name regex_syntax /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/regex-syntax-0.6.18/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="unicode"' --cfg 'feature="unicode-age"' --cfg 'feature="unicode-bool"' --cfg 'feature="unicode-case"' --cfg 'feature="unicode-gencat"' --cfg 'feature="unicode-perl"' --cfg 'feature="unicode-script"' --cfg 'feature="unicode-segment"' -C metadata=604baccf8464f333 -C extra-filename=-604baccf8464f333 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build --edition=2018 /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/ryu-1.0.5/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no -C metadata=a40cc9c191e07da8 -C extra-filename=-a40cc9c191e07da8 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/build/ryu-a40cc9c191e07da8 -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build --edition=2018 /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/getrandom-0.1.15/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="std"' -C metadata=3134d02611660405 -C extra-filename=-3134d02611660405 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/build/getrandom-3134d02611660405 -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build --edition=2018 /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.8.1/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no -C metadata=4f258883be84b941 -C extra-filename=-4f258883be84b941 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/build/rayon-core-4f258883be84b941 -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build --edition=2018 /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_json-1.0.57/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=9c7f2a71de758875 -C extra-filename=-9c7f2a71de758875 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/build/serde_json-9c7f2a71de758875 -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name smallvec --edition=2018 /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/smallvec-1.4.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=af516ba081f6df94 -C extra-filename=-af516ba081f6df94 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name itoa /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/itoa-0.4.6/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=def6b42508610d1c -C extra-filename=-def6b42508610d1c --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build --edition=2018 /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/inventory-0.1.9/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no -C metadata=55eb92d7e72d18d1 -C extra-filename=-55eb92d7e72d18d1 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/build/inventory-55eb92d7e72d18d1 -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name proc_macro_hack --edition=2018 /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro-hack-0.5.18/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -Cembed-bitcode=no -C metadata=24f8c9a7698fc568 -C extra-filename=-24f8c9a7698fc568 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --extern proc_macro --cap-lints allow`
       Running `rustc --crate-name pkg_config /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/pkg-config-0.3.18/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=a729ffec8f42b1bf -C extra-filename=-a729ffec8f42b1bf --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/bitflags-1.2.1/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' -C metadata=86d2212697398c07 -C extra-filename=-86d2212697398c07 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/build/bitflags-86d2212697398c07 -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name cc --edition=2018 /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/cc-1.0.60/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=bd7ffcf8ae7a9c20 -C extra-filename=-bd7ffcf8ae7a9c20 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
     Compiling unindent v0.1.6
     Compiling version_check v0.9.2
     Compiling ppv-lite86 v0.2.9
     Compiling number_prefix v0.3.0
     Compiling strsim v0.8.0
     Compiling vec_map v0.8.2
     Compiling ansi_term v0.11.0
     Compiling unicode_categories v0.1.1
       Running `rustc --crate-name unicode_width /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-width-0.1.8/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' -C metadata=2ffe7097d8c6b666 -C extra-filename=-2ffe7097d8c6b666 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name either /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/either-1.6.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="use_std"' -C metadata=644a45e467402f81 -C extra-filename=-644a45e467402f81 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name version_check /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/version_check-0.9.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=aa50462cc4c9df50 -C extra-filename=-aa50462cc4c9df50 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name unindent --edition=2018 /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/unindent-0.1.6/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=fdeaf6996f560ff0 -C extra-filename=-fdeaf6996f560ff0 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name ppv_lite86 --edition=2018 /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.9/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="simd"' --cfg 'feature="std"' -C metadata=e3e8e9d2c7899d24 -C extra-filename=-e3e8e9d2c7899d24 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name number_prefix /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/number_prefix-0.3.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=a640ea83003307f7 -C extra-filename=-a640ea83003307f7 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name strsim /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/strsim-0.8.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=816b20067865d64c -C extra-filename=-816b20067865d64c --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name vec_map /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/vec_map-0.8.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=a7a30dfbdcea21f0 -C extra-filename=-a7a30dfbdcea21f0 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name ansi_term /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/ansi_term-0.11.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=9c09db9f9cbc7749 -C extra-filename=-9c09db9f9cbc7749 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name unicode_categories /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode_categories-0.1.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=f5d72f9ccd926082 -C extra-filename=-f5d72f9ccd926082 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --cap-lints allow`
     Compiling lock_api v0.3.4
       Running `rustc --crate-name lock_api --edition=2018 /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/lock_api-0.3.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no --cfg 'feature="nightly"' -C metadata=54cc9296368f9d0e -C extra-filename=-54cc9296368f9d0e --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --extern scopeguard=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps/libscopeguard-91afa33e60eb09b1.rmeta --cap-lints allow`
     Compiling thread_local v1.0.1
       Running `rustc --crate-name thread_local /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/thread_local-1.0.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=44b3f6e675105288 -C extra-filename=-44b3f6e675105288 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --extern lazy_static=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps/liblazy_static-09f05f31cfc64306.rmeta --cap-lints allow`
     Compiling textwrap v0.11.0
       Running `rustc --crate-name textwrap /home/aarbelle/.cargo/registry/src/github.com-1ecc6299db9ec823/textwrap-0.11.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -Cembed-bitcode=no -C metadata=05dca2f2bb6ce7b5 -C extra-filename=-05dca2f2bb6ce7b5 --out-dir /tmp/pip-install-ohxny31i/tokenizers/target/release/deps -L dependency=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps --extern unicode_width=/tmp/pip-install-ohxny31i/tokenizers/target/release/deps/libunicode_width-2ffe7097d8c6b666.rmeta --cap-lints allow`
       Running `/tmp/pip-install-ohxny31i/tokenizers/target/release/build/serde_json-9c7f2a71de758875/build-script-build`
       Running `/tmp/pip-install-ohxny31i/tokenizers/target/release/build/rayon-core-4f258883be84b941/build-script-build`
  error: failed to run custom build command for `serde_json v1.0.57`
  
  Caused by:
    could not execute process `/tmp/pip-install-ohxny31i/tokenizers/target/release/build/serde_json-9c7f2a71de758875/build-script-build` (never executed)
  
  Caused by:
    No such file or directory (os error 2)
  warning: build failed, waiting for other jobs to finish...
  error: failed to run custom build command for `rayon-core v1.8.1`
  
  Caused by:
    could not execute process `/tmp/pip-install-ohxny31i/tokenizers/target/release/build/rayon-core-4f258883be84b941/build-script-build` (never executed)
  
  Caused by:
    No such file or directory (os error 2)
  warning: build failed, waiting for other jobs to finish...
  error: build failed
  /tmp/pip-build-env-7kdpvzfy/overlay/lib/python3.6/site-packages/setuptools/dist.py:452: UserWarning: Normalizing '0.8.1.rc2' to '0.8.1rc2'
    warnings.warn(tmpl.format(**locals()))
  cargo rustc --lib --manifest-path Cargo.toml --features pyo3/extension-module --release --verbose -- --crate-type cdylib
  error: cargo failed with code: 101
  
  ----------------------------------------
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed directly
```

@arbellea Please make an issue on the tokenizer page. https://github.com/huggingface/tokenizers

The solution is here: https://github.com/huggingface/tokenizers/issues/431

`sudo pip3 install transformers --no-binary :all:` worked for me

conda install -c huggingface transformers


Use This it will work for sure (M1 also) no need for rust if u get sure try rust and then this in your specific env

Installing Rust via `homebrew` did it for me on a **_Mac OS Monterey M1 Silicon_**
- Python 3.9.9
- pip 21.3.1

Using a conda environment..
```
brew install rustup
rustup-init
source ~/.cargo/env
rustc --version
pip install tokenizers
```

> I was having the same issue on virtualenv over Mac OS Mojave. Managed to solve it and install Transformers 2.5.1 by manually install the last version of tokenizers (0.6.0) instead of 0.5.2 that is required in the transformer package.
> 
> pip install tokenizers
> 
> Git clone latest version of transformers:
> 
> git clone https://github.com/huggingface/transformers
> 
> Before running the installation edit transformers/setup.py and change requirement of tokenizers to 0.6.0
> 
> Line 93: install_requires=[ "numpy", "tokenizers == 0.6.0",
> 
> Then run as usual:
> 
> cd transformers pip install .
> 
> I assume that you could also skip the first step and just collect the package as you run the install. I'm quite new to this, so just wanted to share my take.

This resolved my issue.

> 

Solved this problem by using python=3.7.9 instead of python=3.6.7 in conda env

> > I managed to solve the issue by installing Rust compiler
> > 
> > * Install Rust [link](https://www.rust-lang.org/tools/install) `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`
> > * Restart the terminal
> > * `pip install transformers==2.5.1`
> 
> It works for me, thanks! You can do `source $HOME/.cargo/env` instead of restarting the terminal.


Using  an M1 macbook here. This solved the issue for me, thanks a ton!

> I managed to solve the issue by installing Rust compiler
> 
> * Install Rust [link](https://www.rust-lang.org/tools/install) `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`
> * Restart the terminal
> * `pip install transformers==2.5.1`

Successfully built tokenizers
Failed to build sentencepiece
Installing collected packages: tokenizers, sentencepiece, certifi, urllib3, tqdm, regex, jmespath, idna, filelock, click, charset-normalizer, sacremoses, requests, botocore, s3transfer, boto3, transformers
  Running setup.py install for sentencepiece ... error

> I managed to solve the issue by installing Rust compiler
> 
> * Install Rust [link](https://www.rust-lang.org/tools/install) `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`
> * `pip install transformers==2.5.1`

Only solution you need

I'm facing the issue in Vercel serverless deployments, and the problem is I'cant install rust there? Any other solution?
Python version is 3.9

> Hi, I had the same problem and resolved it by installing rust. "error: Can not find Rust compiler"
> 
> For MacOS, I used "curl https://sh.rustup.rs -sSf | sh". I also found that it needed a nightly version of rust, so you have to specify that in the install options.

Same issue, missing Rust compiler. This command fixed it

 Rust compiler, fixed this for me.
 
  curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

> > 
> 
> Solved this problem by using python=3.7.9 instead of python=3.6.7 in conda env

worked for me on windows10

> I'm facing the issue in Vercel serverless deployments, and the problem is I'cant install rust there? Any other solution?
> Python version is 3.9

@akashp1712 facing the same issue is a deployment context, where it's not possible to install rust. Did you find a solution?

> > I'm facing the issue in Vercel serverless deployments, and the problem is I'cant install rust there? Any other solution?
> > Python version is 3.9
> 
> @akashp1712 facing the same issue is a deployment context, where it's not possible to install rust. Did you find a solution?

Yes @bfelbo, I tried the below in requirements.txt and it worked but couldn't use it as Vercel have hard dependency of 150MB build.

**requirements.txt**
```
nltk
-f https://download.pytorch.org/whl/torch_stable.html
torch==1.11.0+cpu
transformers==2.8.0
```

Mac M1 Monterry.
`pip install transformers==**` does not work for my site.
`conda install transformers` works well.

I have same problem how to solve?

I have installed rust from here and now have different error

windows 10

Python 3.9.9 (tags/v3.9.9:ccb0e6a, Nov 15 2021, 18:08:50) [MSC v.1929 64 bit (AMD64)] on win32


https://github.com/xashru/punctuation-restoration/blob/master/requirements.txt

https://www.rust-lang.org/tools/install

```

C:\punctuation-restoration>pip install -r requirements.txt
Collecting transformers==v2.11.0
  Using cached transformers-2.11.0-py3-none-any.whl (674 kB)
Collecting pytorch-crf
  Using cached pytorch_crf-0.7.2-py3-none-any.whl (9.5 kB)
Requirement already satisfied: packaging in c:\python399\lib\site-packages (from transformers==v2.11.0->-r requirements.txt (line 1)) (21.3)
Requirement already satisfied: requests in c:\python399\lib\site-packages (from transformers==v2.11.0->-r requirements.txt (line 1)) (2.21.0)
Collecting sacremoses
  Using cached sacremoses-0.0.53.tar.gz (880 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: sentencepiece in c:\python399\lib\site-packages (from transformers==v2.11.0->-r requirements.txt (line 1)) (0.1.97)
Requirement already satisfied: regex!=2019.12.17 in c:\python399\lib\site-packages (from transformers==v2.11.0->-r requirements.txt (line 1)) (2022.9.13)
Requirement already satisfied: tqdm>=4.27 in c:\python399\lib\site-packages (from transformers==v2.11.0->-r requirements.txt (line 1)) (4.64.1)
Requirement already satisfied: numpy in c:\python399\lib\site-packages (from transformers==v2.11.0->-r requirements.txt (line 1)) (1.23.3)
Collecting tokenizers==0.7.0
  Using cached tokenizers-0.7.0.tar.gz (81 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: filelock in c:\python399\lib\site-packages (from transformers==v2.11.0->-r requirements.txt (line 1)) (3.8.0)
Requirement already satisfied: colorama in c:\python399\lib\site-packages (from tqdm>=4.27->transformers==v2.11.0->-r requirements.txt (line 1)) (0.4.5)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in c:\python399\lib\site-packages (from packaging->transformers==v2.11.0->-r requirements.txt (line 1)) (3.0.9)
Requirement already satisfied: idna<2.9,>=2.5 in c:\python399\lib\site-packages (from requests->transformers==v2.11.0->-r requirements.txt (line 1)) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in c:\python399\lib\site-packages (from requests->transformers==v2.11.0->-r requirements.txt (line 1)) (2022.9.14)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in c:\python399\lib\site-packages (from requests->transformers==v2.11.0->-r requirements.txt (line 1)) (1.24.3)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:\python399\lib\site-packages (from requests->transformers==v2.11.0->-r requirements.txt (line 1)) (3.0.4)
Requirement already satisfied: six in c:\python399\lib\site-packages (from sacremoses->transformers==v2.11.0->-r requirements.txt (line 1)) (1.12.0)
Collecting click
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting joblib
  Using cached joblib-1.2.0-py3-none-any.whl (297 kB)
Building wheels for collected packages: tokenizers
  Building wheel for tokenizers (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for tokenizers (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [258 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build\lib.win-amd64-cpython-39
      creating build\lib.win-amd64-cpython-39\tokenizers
      copying tokenizers\__init__.py -> build\lib.win-amd64-cpython-39\tokenizers
      creating build\lib.win-amd64-cpython-39\tokenizers\models
      copying tokenizers\models\__init__.py -> build\lib.win-amd64-cpython-39\tokenizers\models
      creating build\lib.win-amd64-cpython-39\tokenizers\decoders
      copying tokenizers\decoders\__init__.py -> build\lib.win-amd64-cpython-39\tokenizers\decoders
      creating build\lib.win-amd64-cpython-39\tokenizers\normalizers
      copying tokenizers\normalizers\__init__.py -> build\lib.win-amd64-cpython-39\tokenizers\normalizers
      creating build\lib.win-amd64-cpython-39\tokenizers\pre_tokenizers
      copying tokenizers\pre_tokenizers\__init__.py -> build\lib.win-amd64-cpython-39\tokenizers\pre_tokenizers
      creating build\lib.win-amd64-cpython-39\tokenizers\processors
      copying tokenizers\processors\__init__.py -> build\lib.win-amd64-cpython-39\tokenizers\processors
      creating build\lib.win-amd64-cpython-39\tokenizers\trainers
      copying tokenizers\trainers\__init__.py -> build\lib.win-amd64-cpython-39\tokenizers\trainers
      creating build\lib.win-amd64-cpython-39\tokenizers\implementations
      copying tokenizers\implementations\base_tokenizer.py -> build\lib.win-amd64-cpython-39\tokenizers\implementations
      copying tokenizers\implementations\bert_wordpiece.py -> build\lib.win-amd64-cpython-39\tokenizers\implementations
      copying tokenizers\implementations\byte_level_bpe.py -> build\lib.win-amd64-cpython-39\tokenizers\implementations
      copying tokenizers\implementations\char_level_bpe.py -> build\lib.win-amd64-cpython-39\tokenizers\implementations
      copying tokenizers\implementations\sentencepiece_bpe.py -> build\lib.win-amd64-cpython-39\tokenizers\implementations
      copying tokenizers\implementations\__init__.py -> build\lib.win-amd64-cpython-39\tokenizers\implementations
      copying tokenizers\__init__.pyi -> build\lib.win-amd64-cpython-39\tokenizers
      copying tokenizers\models\__init__.pyi -> build\lib.win-amd64-cpython-39\tokenizers\models
      copying tokenizers\decoders\__init__.pyi -> build\lib.win-amd64-cpython-39\tokenizers\decoders
      copying tokenizers\normalizers\__init__.pyi -> build\lib.win-amd64-cpython-39\tokenizers\normalizers
      copying tokenizers\pre_tokenizers\__init__.pyi -> build\lib.win-amd64-cpython-39\tokenizers\pre_tokenizers
      copying tokenizers\processors\__init__.pyi -> build\lib.win-amd64-cpython-39\tokenizers\processors
      copying tokenizers\trainers\__init__.pyi -> build\lib.win-amd64-cpython-39\tokenizers\trainers
      running build_ext
      running build_rust
      info: syncing channel updates for 'nightly-x86_64-pc-windows-msvc'
      info: latest update on 2022-10-18, rust version 1.66.0-nightly (06f049a35 2022-10-17)
      info: downloading component 'cargo'
      info: downloading component 'clippy'
      info: downloading component 'rust-docs'
      info: downloading component 'rust-std'
      info: downloading component 'rustc'
      info: downloading component 'rustfmt'
      info: installing component 'cargo'
      info: installing component 'clippy'
      info: installing component 'rust-docs'
      info: retrying renaming 'C:\Users\King\.rustup\tmp\qkopjzc7kj3elau0_dir\rust-docs\share/doc/rust/html' to 'C:\Users\King\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\share/doc/rust/html'
      info: installing component 'rust-std'
      info: installing component 'rustc'
      info: installing component 'rustfmt'
      cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module -- --crate-type cdylib
      warning: unused manifest key: target.x86_64-apple-darwin.rustflags
          Updating crates.io index
       Downloading crates ...
        Downloaded autocfg v1.1.0
        Downloaded serde v1.0.145
        Downloaded serde_json v1.0.86
        Downloaded rand v0.7.3
        Downloaded itoa v1.0.4
        Downloaded memchr v2.5.0
        Downloaded num_cpus v1.13.1
        Downloaded inventory-impl v0.1.11
        Downloaded parking_lot v0.10.2
        Downloaded parking_lot_core v0.7.2
        Downloaded number_prefix v0.3.0
        Downloaded indoc-impl v0.3.6
        Downloaded serde_derive v1.0.145
        Downloaded ryu v1.0.11
        Downloaded version_check v0.9.4
        Downloaded unicode-width v0.1.10
        Downloaded syn v1.0.102
        Downloaded ghost v0.1.6
        Downloaded lock_api v0.3.4
        Downloaded inventory v0.1.11
        Downloaded indicatif v0.14.0
        Downloaded getrandom v0.1.16
        Downloaded textwrap v0.11.0
        Downloaded lazy_static v1.4.0
        Downloaded ctor v0.1.24
        Downloaded crossbeam-utils v0.8.12
        Downloaded num-traits v0.2.15
        Downloaded unindent v0.1.10
        Downloaded unicode-normalization-alignments v0.1.12
        Downloaded pyo3cls v0.9.2
        Downloaded pyo3 v0.9.2
        Downloaded paste-impl v0.1.18
        Downloaded rand_core v0.5.1
        Downloaded strsim v0.8.0
        Downloaded scopeguard v1.1.0
        Downloaded vec_map v0.8.2
        Downloaded unicode-ident v1.0.5
        Downloaded rayon-core v1.9.3
        Downloaded rayon v1.5.3
        Downloaded unicode_categories v0.1.1
        Downloaded pyo3-derive-backend v0.9.2
        Downloaded bitflags v1.3.2
        Downloaded cfg-if v1.0.0
        Downloaded cfg-if v0.1.10
        Downloaded atty v0.2.14
        Downloaded regex v1.6.0
        Downloaded memoffset v0.6.5
        Downloaded indoc v0.3.6
        Downloaded either v1.8.0
        Downloaded crossbeam-deque v0.8.2
        Downloaded crossbeam-channel v0.5.6
        Downloaded clap v2.34.0
        Downloaded regex-syntax v0.6.27
        Downloaded winapi v0.3.9
        Downloaded libc v0.2.135
        Downloaded aho-corasick v0.7.19
        Downloaded console v0.15.2
        Downloaded crossbeam-epoch v0.9.11
        Downloaded encode_unicode v0.3.6
        Downloaded smallvec v1.10.0
        Downloaded terminal_size v0.1.17
        Downloaded proc-macro2 v1.0.47
        Downloaded quote v1.0.21
        Downloaded rand_chacha v0.2.2
        Downloaded proc-macro-hack v0.5.19
        Downloaded ppv-lite86 v0.2.16
        Downloaded paste v0.1.18
         Compiling proc-macro2 v1.0.47
         Compiling quote v1.0.21
         Compiling unicode-ident v1.0.5
         Compiling syn v1.0.102
         Compiling autocfg v1.1.0
         Compiling memchr v2.5.0
         Compiling winapi v0.3.9
         Compiling cfg-if v1.0.0
         Compiling serde_derive v1.0.145
         Compiling serde v1.0.145
         Compiling serde_json v1.0.86
         Compiling scopeguard v1.1.0
         Compiling crossbeam-utils v0.8.12
         Compiling proc-macro-hack v0.5.19
         Compiling getrandom v0.1.16
         Compiling libc v0.2.135
           Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-macro2-1.0.47\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"proc-macro\"" -C metadata=5b0e58d159021849 -C extra-filename=-5b0e58d159021849 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\proc-macro2-5b0e58d159021849 -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\quote-1.0.21\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"proc-macro\"" -C metadata=142419b9cedb1d17 -C extra-filename=-142419b9cedb1d17 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\quote-142419b9cedb1d17 -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name unicode_ident --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\unicode-ident-1.0.5\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off -C metadata=1afecb6243c40eff -C extra-filename=-1afecb6243c40eff --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\syn-1.0.102\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"clone-impls\"" --cfg "feature=\"default\"" --cfg "feature=\"derive\"" --cfg "feature=\"extra-traits\"" --cfg "feature=\"full\"" --cfg "feature=\"parsing\"" --cfg "feature=\"printing\"" --cfg "feature=\"proc-macro\"" --cfg "feature=\"quote\"" -C metadata=b8e33a9e5e50652b -C extra-filename=-b8e33a9e5e50652b --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\syn-b8e33a9e5e50652b -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name autocfg C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\autocfg-1.1.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off -C metadata=572016c50f479faf -C extra-filename=-572016c50f479faf --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\memchr-2.5.0\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=4ba3537c9d483825 -C extra-filename=-4ba3537c9d483825 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\memchr-4ba3537c9d483825 -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name build_script_build C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\winapi-0.3.9\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"consoleapi\"" --cfg "feature=\"errhandlingapi\"" --cfg "feature=\"fileapi\"" --cfg "feature=\"handleapi\"" --cfg "feature=\"minwinbase\"" --cfg "feature=\"minwindef\"" --cfg "feature=\"ntstatus\"" --cfg "feature=\"processenv\"" --cfg "feature=\"winbase\"" --cfg "feature=\"wincon\"" --cfg "feature=\"winerror\"" --cfg "feature=\"winnt\"" --cfg "feature=\"winuser\"" -C metadata=9ff0f262099ebbbd -C extra-filename=-9ff0f262099ebbbd --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\winapi-9ff0f262099ebbbd -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name cfg_if --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\cfg-if-1.0.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=360d109d20cdf1de -C extra-filename=-360d109d20cdf1de --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name build_script_build C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\serde_derive-1.0.145\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" -C metadata=84c8f4354cf6545a -C extra-filename=-84c8f4354cf6545a --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\serde_derive-84c8f4354cf6545a -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name build_script_build C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\serde-1.0.145\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"derive\"" --cfg "feature=\"serde_derive\"" --cfg "feature=\"std\"" -C metadata=1080dbca61ce40eb -C extra-filename=-1080dbca61ce40eb --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\serde-1080dbca61ce40eb -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\serde_json-1.0.86\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=b3ba47c415a7892f -C extra-filename=-b3ba47c415a7892f --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\serde_json-b3ba47c415a7892f -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name scopeguard C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\scopeguard-1.1.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=38978501dffc7e67 -C extra-filename=-38978501dffc7e67 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\crossbeam-utils-0.8.12\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=66c260199ab3a6b1 -C extra-filename=-66c260199ab3a6b1 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\crossbeam-utils-66c260199ab3a6b1 -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-macro-hack-0.5.19\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off -C metadata=43fb356fc25df1f2 -C extra-filename=-43fb356fc25df1f2 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\proc-macro-hack-43fb356fc25df1f2 -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\getrandom-0.1.16\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"std\"" -C metadata=c55ade8324d95812 -C extra-filename=-c55ade8324d95812 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\getrandom-c55ade8324d95812 -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name build_script_build C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\libc-0.2.135\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=e99f7f26a77660cd -C extra-filename=-e99f7f26a77660cd --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\libc-e99f7f26a77660cd -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
         Compiling unicode-width v0.1.10
         Compiling rayon-core v1.9.3
           Running `rustc --crate-name unicode_width C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\unicode-width-0.1.10\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"default\"" -C metadata=5f855d5352d382a0 -C extra-filename=-5f855d5352d382a0 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\rayon-core-1.9.3\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off -C metadata=e7c93bf382490bd0 -C extra-filename=-e7c93bf382490bd0 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\rayon-core-e7c93bf382490bd0 -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
         Compiling smallvec v1.10.0
           Running `rustc --crate-name smallvec --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\smallvec-1.10.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=881054317b8f4d9c -C extra-filename=-881054317b8f4d9c --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
         Compiling ryu v1.0.11
           Running `rustc --crate-name ryu --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\ryu-1.0.11\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off -C metadata=1d25c68059b095ff -C extra-filename=-1d25c68059b095ff --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
         Compiling memoffset v0.6.5
         Compiling crossbeam-epoch v0.9.11
           Running `rustc --crate-name build_script_build C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\memoffset-0.6.5\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" -C metadata=8550414a9ea9eb93 -C extra-filename=-8550414a9ea9eb93 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\memoffset-8550414a9ea9eb93 -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern autocfg=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libautocfg-572016c50f479faf.rlib --cap-lints allow`
           Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\crossbeam-epoch-0.9.11\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"alloc\"" --cfg "feature=\"std\"" -C metadata=a5f96eb43b93974a -C extra-filename=-a5f96eb43b93974a --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\crossbeam-epoch-a5f96eb43b93974a -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern autocfg=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libautocfg-572016c50f479faf.rlib --cap-lints allow`
         Compiling regex-syntax v0.6.27
           Running `rustc --crate-name regex_syntax --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\regex-syntax-0.6.27\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"unicode\"" --cfg "feature=\"unicode-age\"" --cfg "feature=\"unicode-bool\"" --cfg "feature=\"unicode-case\"" --cfg "feature=\"unicode-gencat\"" --cfg "feature=\"unicode-perl\"" --cfg "feature=\"unicode-script\"" --cfg "feature=\"unicode-segment\"" -C metadata=ecd6baa27c468f0d -C extra-filename=-ecd6baa27c468f0d --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\syn-b8e33a9e5e50652b\build-script-build`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\proc-macro2-5b0e58d159021849\build-script-build`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\quote-142419b9cedb1d17\build-script-build`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\winapi-9ff0f262099ebbbd\build-script-build`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\serde_derive-84c8f4354cf6545a\build-script-build`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\memoffset-8550414a9ea9eb93\build-script-build`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\proc-macro-hack-43fb356fc25df1f2\build-script-build`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\getrandom-c55ade8324d95812\build-script-build`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\libc-e99f7f26a77660cd\build-script-build`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\serde-1080dbca61ce40eb\build-script-build`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\crossbeam-epoch-a5f96eb43b93974a\build-script-build`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\crossbeam-utils-66c260199ab3a6b1\build-script-build`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\memchr-4ba3537c9d483825\build-script-build`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\memchr-4ba3537c9d483825\build-script-build`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\rayon-core-e7c93bf382490bd0\build-script-build`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\serde_json-b3ba47c415a7892f\build-script-build`
           Running `rustc --crate-name proc_macro2 --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-macro2-1.0.47\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"proc-macro\"" -C metadata=30944198976f606e -C extra-filename=-30944198976f606e --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern unicode_ident=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libunicode_ident-1afecb6243c40eff.rmeta --cap-lints allow --cfg use_proc_macro --cfg wrap_proc_macro --cfg proc_macro_span`
         Compiling itoa v1.0.4
           Running `rustc --crate-name itoa --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\itoa-1.0.4\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off -C metadata=c5e72f9360d1cff0 -C extra-filename=-c5e72f9360d1cff0 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name winapi C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\winapi-0.3.9\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"consoleapi\"" --cfg "feature=\"errhandlingapi\"" --cfg "feature=\"fileapi\"" --cfg "feature=\"handleapi\"" --cfg "feature=\"minwinbase\"" --cfg "feature=\"minwindef\"" --cfg "feature=\"ntstatus\"" --cfg "feature=\"processenv\"" --cfg "feature=\"winbase\"" --cfg "feature=\"wincon\"" --cfg "feature=\"winerror\"" --cfg "feature=\"winnt\"" --cfg "feature=\"winuser\"" -C metadata=c43802643be2e565 -C extra-filename=-c43802643be2e565 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow -l dylib=advapi32 -l dylib=cfgmgr32 -l dylib=gdi32 -l dylib=kernel32 -l dylib=msimg32 -l dylib=opengl32 -l dylib=user32 -l dylib=winspool --cfg "feature=\"vadefs\"" --cfg "feature=\"cfgmgr32\"" --cfg "feature=\"devpropdef\"" --cfg "feature=\"processthreadsapi\"" --cfg "feature=\"basetsd\"" --cfg "feature=\"winreg\"" --cfg "feature=\"ntdef\"" --cfg "feature=\"libloaderapi\"" --cfg "feature=\"wincontypes\"" --cfg "feature=\"vcruntime\"" --cfg "feature=\"excpt\"" --cfg "feature=\"wtypesbase\"" --cfg "feature=\"reason\"" --cfg "feature=\"limits\"" --cfg "feature=\"rpcndr\"" --cfg "feature=\"ktmtypes\"" --cfg "feature=\"guiddef\"" --cfg "feature=\"windef\"" --cfg "feature=\"wingdi\"" --cfg "feature=\"cfg\""`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\serde-1080dbca61ce40eb\build-script-build`
           Running `rustc --crate-name memoffset C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\memoffset-0.6.5\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"default\"" -C metadata=d521e9d5ca1cfb74 -C extra-filename=-d521e9d5ca1cfb74 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow --cfg tuple_ty --cfg allow_clippy --cfg maybe_uninit --cfg doctests --cfg raw_ref_macros`
           Running `rustc --crate-name proc_macro_hack --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-macro-hack-0.5.19\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debug-assertions=off -C metadata=097737d8e2c954ab -C extra-filename=-097737d8e2c954ab --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern proc_macro --cap-lints allow`
           Running `rustc --crate-name getrandom --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\getrandom-0.1.16\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"std\"" -C metadata=6c7f93dfc6444a9c -C extra-filename=-6c7f93dfc6444a9c --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern cfg_if=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcfg_if-360d109d20cdf1de.rmeta --cap-lints allow -l advapi32`
           Running `rustc --crate-name libc C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\libc-0.2.135\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=86a1bab4b120f6a2 -C extra-filename=-86a1bab4b120f6a2 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow --cfg freebsd11 --cfg libc_priv_mod_use --cfg libc_union --cfg libc_const_size_of --cfg libc_align --cfg libc_int128 --cfg libc_core_cvoid --cfg libc_packedN --cfg libc_cfg_target_vendor --cfg libc_non_exhaustive --cfg libc_ptr_addr_of --cfg libc_underscore_const_names --cfg libc_const_extern_fn`
         Compiling rayon v1.5.3
           Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\rayon-1.5.3\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off -C metadata=f6ea986aed560875 -C extra-filename=-f6ea986aed560875 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\rayon-f6ea986aed560875 -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern autocfg=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libautocfg-572016c50f479faf.rlib --cap-lints allow`
         Compiling num-traits v0.2.15
           Running `rustc --crate-name build_script_build C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\num-traits-0.2.15\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=286ff2921bdb81ee -C extra-filename=-286ff2921bdb81ee --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\num-traits-286ff2921bdb81ee -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern autocfg=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libautocfg-572016c50f479faf.rlib --cap-lints allow`
           Running `rustc --crate-name crossbeam_utils --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\crossbeam-utils-0.8.12\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=3869c69ecfdeae93 -C extra-filename=-3869c69ecfdeae93 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern cfg_if=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcfg_if-360d109d20cdf1de.rmeta --cap-lints allow`
           Running `rustc --crate-name memchr --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\memchr-2.5.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=055ae21494443b01 -C extra-filename=-055ae21494443b01 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow --cfg memchr_runtime_simd --cfg memchr_runtime_sse2 --cfg memchr_runtime_sse42 --cfg memchr_runtime_avx`
           Running `rustc --crate-name memchr --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\memchr-2.5.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=f54a1ac40e150573 -C extra-filename=-f54a1ac40e150573 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow --cfg memchr_runtime_simd --cfg memchr_runtime_sse2 --cfg memchr_runtime_sse42 --cfg memchr_runtime_avx`
         Compiling unindent v0.1.10
           Running `rustc --crate-name unindent --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\unindent-0.1.10\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off -C metadata=fcf514872b9c9d8e -C extra-filename=-fcf514872b9c9d8e --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name regex_syntax --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\regex-syntax-0.6.27\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"default\"" --cfg "feature=\"unicode\"" --cfg "feature=\"unicode-age\"" --cfg "feature=\"unicode-bool\"" --cfg "feature=\"unicode-case\"" --cfg "feature=\"unicode-gencat\"" --cfg "feature=\"unicode-perl\"" --cfg "feature=\"unicode-script\"" --cfg "feature=\"unicode-segment\"" -C metadata=5a236c973584f460 -C extra-filename=-5a236c973584f460 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
         Compiling cfg-if v0.1.10
           Running `rustc --crate-name cfg_if --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\cfg-if-0.1.10\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=f3af8d70d946c761 -C extra-filename=-f3af8d70d946c761 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
         Compiling encode_unicode v0.3.6
           Running `rustc --crate-name encode_unicode C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\encode_unicode-0.3.6\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=3130f1c305b48f53 -C extra-filename=-3130f1c305b48f53 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
         Compiling inventory v0.1.11
         Compiling ppv-lite86 v0.2.16
           Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\inventory-0.1.11\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off -C metadata=2a48a1911251848c -C extra-filename=-2a48a1911251848c --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\inventory-2a48a1911251848c -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name ppv_lite86 --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\ppv-lite86-0.2.16\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"simd\"" --cfg "feature=\"std\"" -C metadata=94c9eb93ee689d3b -C extra-filename=-94c9eb93ee689d3b --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
         Compiling rand_core v0.5.1
           Running `rustc --crate-name rand_core --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\rand_core-0.5.1\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"alloc\"" --cfg "feature=\"getrandom\"" --cfg "feature=\"std\"" -C metadata=06e7d83191242dc6 -C extra-filename=-06e7d83191242dc6 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern getrandom=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libgetrandom-6c7f93dfc6444a9c.rmeta --cap-lints allow`
         Compiling version_check v0.9.4
           Running `rustc --crate-name version_check C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\version_check-0.9.4\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off -C metadata=0584bbe3f7491e0d -C extra-filename=-0584bbe3f7491e0d --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name quote --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\quote-1.0.21\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"proc-macro\"" -C metadata=dabbd39e3909793b -C extra-filename=-dabbd39e3909793b --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern proc_macro2=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libproc_macro2-30944198976f606e.rmeta --cap-lints allow`
         Compiling num_cpus v1.13.1
           Running `rustc --crate-name num_cpus C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\num_cpus-1.13.1\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=31ad845e15e2b15d -C extra-filename=-31ad845e15e2b15d --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
         Compiling lazy_static v1.4.0
           Running `rustc --crate-name lazy_static C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\lazy_static-1.4.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=431f071606c587f4 -C extra-filename=-431f071606c587f4 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
           Running `rustc --crate-name crossbeam_epoch --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\crossbeam-epoch-0.9.11\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"alloc\"" --cfg "feature=\"std\"" -C metadata=4897f65d76bcc39c -C extra-filename=-4897f65d76bcc39c --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern cfg_if=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcfg_if-360d109d20cdf1de.rmeta --extern crossbeam_utils=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcrossbeam_utils-3869c69ecfdeae93.rmeta --extern memoffset=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libmemoffset-d521e9d5ca1cfb74.rmeta --extern scopeguard=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libscopeguard-38978501dffc7e67.rmeta --cap-lints allow`
         Compiling crossbeam-channel v0.5.6
           Running `rustc --crate-name crossbeam_channel --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\crossbeam-channel-0.5.6\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"crossbeam-utils\"" --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=658fbd4d6c655a6f -C extra-filename=-658fbd4d6c655a6f --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern cfg_if=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcfg_if-360d109d20cdf1de.rmeta --extern crossbeam_utils=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcrossbeam_utils-3869c69ecfdeae93.rmeta --cap-lints allow`
           Running `rustc --crate-name syn --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\syn-1.0.102\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"clone-impls\"" --cfg "feature=\"default\"" --cfg "feature=\"derive\"" --cfg "feature=\"extra-traits\"" --cfg "feature=\"full\"" --cfg "feature=\"parsing\"" --cfg "feature=\"printing\"" --cfg "feature=\"proc-macro\"" --cfg "feature=\"quote\"" -C metadata=ed20b2b04ea8255e -C extra-filename=-ed20b2b04ea8255e --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern proc_macro2=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libproc_macro2-30944198976f606e.rmeta --extern quote=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libquote-dabbd39e3909793b.rmeta --extern unicode_ident=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libunicode_ident-1afecb6243c40eff.rmeta --cap-lints allow`
         Compiling aho-corasick v0.7.19
           Running `rustc --crate-name aho_corasick --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\aho-corasick-0.7.19\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=f1529ca83e28da6f -C extra-filename=-f1529ca83e28da6f --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern memchr=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libmemchr-055ae21494443b01.rmeta --cap-lints allow`
         Compiling rand_chacha v0.2.2
           Running `rustc --crate-name aho_corasick --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\aho-corasick-0.7.19\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=dfcbc49062ee55e6 -C extra-filename=-dfcbc49062ee55e6 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern memchr=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libmemchr-f54a1ac40e150573.rmeta --cap-lints allow`
           Running `rustc --crate-name rand_chacha --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\rand_chacha-0.2.2\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"std\"" -C metadata=42edb5e8f22d28e3 -C extra-filename=-42edb5e8f22d28e3 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern ppv_lite86=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libppv_lite86-94c9eb93ee689d3b.rmeta --extern rand_core=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\librand_core-06e7d83191242dc6.rmeta --cap-lints allow`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\inventory-2a48a1911251848c\build-script-build`
         Compiling crossbeam-deque v0.8.2
           Running `rustc --crate-name crossbeam_deque --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\crossbeam-deque-0.8.2\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"crossbeam-epoch\"" --cfg "feature=\"crossbeam-utils\"" --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=b75d3846468b682e -C extra-filename=-b75d3846468b682e --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern cfg_if=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcfg_if-360d109d20cdf1de.rmeta --extern crossbeam_epoch=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcrossbeam_epoch-4897f65d76bcc39c.rmeta --extern crossbeam_utils=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcrossbeam_utils-3869c69ecfdeae93.rmeta --cap-lints allow`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\num-traits-286ff2921bdb81ee\build-script-build`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\rayon-f6ea986aed560875\build-script-build`
           Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\serde_json-b3ba47c415a7892f\build-script-build`
         Compiling textwrap v0.11.0
           Running `rustc --crate-name rayon_core --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\rayon-core-1.9.3\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=0dc7012c06ddf19e -C extra-filename=-0dc7012c06ddf19e --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern crossbeam_channel=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcrossbeam_channel-658fbd4d6c655a6f.rmeta --extern crossbeam_deque=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcrossbeam_deque-b75d3846468b682e.rmeta --extern crossbeam_utils=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcrossbeam_utils-3869c69ecfdeae93.rmeta --extern num_cpus=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libnum_cpus-31ad845e15e2b15d.rmeta --cap-lints allow`
           Running `rustc --crate-name textwrap C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\textwrap-0.11.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=65eec7bbb3cdf0f8 -C extra-filename=-65eec7bbb3cdf0f8 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern unicode_width=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libunicode_width-5f855d5352d382a0.rmeta --cap-lints allow`
         Compiling paste-impl v0.1.18
           Running `rustc --crate-name paste_impl --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\paste-impl-0.1.18\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debug-assertions=off -C metadata=d5e32c087e023b55 -C extra-filename=-d5e32c087e023b55 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern proc_macro_hack=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\proc_macro_hack-097737d8e2c954ab.dll --extern proc_macro --cap-lints allow`
         Compiling lock_api v0.3.4
         Compiling strsim v0.8.0
           Running `rustc --crate-name lock_api --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\lock_api-0.3.4\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"nightly\"" -C metadata=2fe17ee22c58d967 -C extra-filename=-2fe17ee22c58d967 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern scopeguard=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libscopeguard-38978501dffc7e67.rmeta --cap-lints allow`
           Running `rustc --crate-name strsim C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\strsim-0.8.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=8fb9b7eda8bf3ac7 -C extra-filename=-8fb9b7eda8bf3ac7 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
      error[E0557]: feature has been removed
        --> C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\lock_api-0.3.4\src\lib.rs:91:42
         |
      91 | #![cfg_attr(feature = "nightly", feature(const_fn))]
         |                                          ^^^^^^^^ feature has been removed
         |
         = note: split into finer-grained feature gates

      For more information about this error, try `rustc --explain E0557`.
      error: could not compile `lock_api` due to previous error

      Caused by:
        process didn't exit successfully: `rustc --crate-name lock_api --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\lock_api-0.3.4\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"nightly\"" -C metadata=2fe17ee22c58d967 -C extra-filename=-2fe17ee22c58d967 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern scopeguard=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libscopeguard-38978501dffc7e67.rmeta --cap-lints allow` (exit code: 1)
      warning: build failed, waiting for other jobs to finish...
      error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module -- --crate-type cdylib` failed with code 101
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

C:\punctuation-restoration>

```

`pip install transformers>=4.10.0` helps me.

`transformers==4.11.3` causes `Building wheel for tokenizers (pyproject.toml) did not run successfully`

I had the same problem when trying to create a dockerfile. I was able to solve using the solution of this issue.

```
# Get Rust; NOTE: using sh for better compatibility with other base images
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
# Add .cargo/bin to PATH
ENV PATH="/root/.cargo/bin:${PATH}"
RUN  pip install --upgrade pip
```

```sudo apt install rustc cargo``` solved my problem finally

i have the same with my windows 7 and python3.6.8.
so i used pyenv to install python3.7.7:
`pyenv install python3.7.7`
then i made dir 'project' and set there local version of python:
```
mkdir project 
cd project
pyenv local 3.7.7
```
then just installed what i needed:
```
python3 -m pip install --upgrade pip
python3 -m pip install transformers
python3 -m pip install Pillow
python3 -m pip install pytorch torchvision cpuonly -c pytorch
```

have a nice day!

Install rust sdk and reopen terminal / cmd / powershell

I had this error when trying to install a specific version of `transformers`: `pip install transformers==4.15.0`
When I tried to install without specifying the version, it worked: `pip install transformers`


while I am installing transformer==4.4, I am facing issue: Failed to build tokenizers.



Hey @GuruPavanKalyan could you refrain from copy pasting your problem on unrelated issues? You should either create a new issue on the `tokenizers` repo as the failure might be on this side, or try to check if the wheel is available for your custom hardware and setup. 

> I have same problem how to solve?
> 
> I have installed rust from here and now have different error
> 
> windows 10
> 
> Python 3.9.9 (tags/v3.9.9:ccb0e6a, Nov 15 2021, 18:08:50) [MSC v.1929 64 bit (AMD64)] on win32
> 
> https://github.com/xashru/punctuation-restoration/blob/master/requirements.txt
> 
> https://www.rust-lang.org/tools/install
> 
> ```
> 
> C:\punctuation-restoration>pip install -r requirements.txt
> Collecting transformers==v2.11.0
>   Using cached transformers-2.11.0-py3-none-any.whl (674 kB)
> Collecting pytorch-crf
>   Using cached pytorch_crf-0.7.2-py3-none-any.whl (9.5 kB)
> Requirement already satisfied: packaging in c:\python399\lib\site-packages (from transformers==v2.11.0->-r requirements.txt (line 1)) (21.3)
> Requirement already satisfied: requests in c:\python399\lib\site-packages (from transformers==v2.11.0->-r requirements.txt (line 1)) (2.21.0)
> Collecting sacremoses
>   Using cached sacremoses-0.0.53.tar.gz (880 kB)
>   Preparing metadata (setup.py) ... done
> Requirement already satisfied: sentencepiece in c:\python399\lib\site-packages (from transformers==v2.11.0->-r requirements.txt (line 1)) (0.1.97)
> Requirement already satisfied: regex!=2019.12.17 in c:\python399\lib\site-packages (from transformers==v2.11.0->-r requirements.txt (line 1)) (2022.9.13)
> Requirement already satisfied: tqdm>=4.27 in c:\python399\lib\site-packages (from transformers==v2.11.0->-r requirements.txt (line 1)) (4.64.1)
> Requirement already satisfied: numpy in c:\python399\lib\site-packages (from transformers==v2.11.0->-r requirements.txt (line 1)) (1.23.3)
> Collecting tokenizers==0.7.0
>   Using cached tokenizers-0.7.0.tar.gz (81 kB)
>   Installing build dependencies ... done
>   Getting requirements to build wheel ... done
>   Preparing metadata (pyproject.toml) ... done
> Requirement already satisfied: filelock in c:\python399\lib\site-packages (from transformers==v2.11.0->-r requirements.txt (line 1)) (3.8.0)
> Requirement already satisfied: colorama in c:\python399\lib\site-packages (from tqdm>=4.27->transformers==v2.11.0->-r requirements.txt (line 1)) (0.4.5)
> Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in c:\python399\lib\site-packages (from packaging->transformers==v2.11.0->-r requirements.txt (line 1)) (3.0.9)
> Requirement already satisfied: idna<2.9,>=2.5 in c:\python399\lib\site-packages (from requests->transformers==v2.11.0->-r requirements.txt (line 1)) (2.8)
> Requirement already satisfied: certifi>=2017.4.17 in c:\python399\lib\site-packages (from requests->transformers==v2.11.0->-r requirements.txt (line 1)) (2022.9.14)
> Requirement already satisfied: urllib3<1.25,>=1.21.1 in c:\python399\lib\site-packages (from requests->transformers==v2.11.0->-r requirements.txt (line 1)) (1.24.3)
> Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:\python399\lib\site-packages (from requests->transformers==v2.11.0->-r requirements.txt (line 1)) (3.0.4)
> Requirement already satisfied: six in c:\python399\lib\site-packages (from sacremoses->transformers==v2.11.0->-r requirements.txt (line 1)) (1.12.0)
> Collecting click
>   Using cached click-8.1.3-py3-none-any.whl (96 kB)
> Collecting joblib
>   Using cached joblib-1.2.0-py3-none-any.whl (297 kB)
> Building wheels for collected packages: tokenizers
>   Building wheel for tokenizers (pyproject.toml) ... error
>   error: subprocess-exited-with-error
> 
>   × Building wheel for tokenizers (pyproject.toml) did not run successfully.
>   │ exit code: 1
>   ╰─> [258 lines of output]
>       running bdist_wheel
>       running build
>       running build_py
>       creating build
>       creating build\lib.win-amd64-cpython-39
>       creating build\lib.win-amd64-cpython-39\tokenizers
>       copying tokenizers\__init__.py -> build\lib.win-amd64-cpython-39\tokenizers
>       creating build\lib.win-amd64-cpython-39\tokenizers\models
>       copying tokenizers\models\__init__.py -> build\lib.win-amd64-cpython-39\tokenizers\models
>       creating build\lib.win-amd64-cpython-39\tokenizers\decoders
>       copying tokenizers\decoders\__init__.py -> build\lib.win-amd64-cpython-39\tokenizers\decoders
>       creating build\lib.win-amd64-cpython-39\tokenizers\normalizers
>       copying tokenizers\normalizers\__init__.py -> build\lib.win-amd64-cpython-39\tokenizers\normalizers
>       creating build\lib.win-amd64-cpython-39\tokenizers\pre_tokenizers
>       copying tokenizers\pre_tokenizers\__init__.py -> build\lib.win-amd64-cpython-39\tokenizers\pre_tokenizers
>       creating build\lib.win-amd64-cpython-39\tokenizers\processors
>       copying tokenizers\processors\__init__.py -> build\lib.win-amd64-cpython-39\tokenizers\processors
>       creating build\lib.win-amd64-cpython-39\tokenizers\trainers
>       copying tokenizers\trainers\__init__.py -> build\lib.win-amd64-cpython-39\tokenizers\trainers
>       creating build\lib.win-amd64-cpython-39\tokenizers\implementations
>       copying tokenizers\implementations\base_tokenizer.py -> build\lib.win-amd64-cpython-39\tokenizers\implementations
>       copying tokenizers\implementations\bert_wordpiece.py -> build\lib.win-amd64-cpython-39\tokenizers\implementations
>       copying tokenizers\implementations\byte_level_bpe.py -> build\lib.win-amd64-cpython-39\tokenizers\implementations
>       copying tokenizers\implementations\char_level_bpe.py -> build\lib.win-amd64-cpython-39\tokenizers\implementations
>       copying tokenizers\implementations\sentencepiece_bpe.py -> build\lib.win-amd64-cpython-39\tokenizers\implementations
>       copying tokenizers\implementations\__init__.py -> build\lib.win-amd64-cpython-39\tokenizers\implementations
>       copying tokenizers\__init__.pyi -> build\lib.win-amd64-cpython-39\tokenizers
>       copying tokenizers\models\__init__.pyi -> build\lib.win-amd64-cpython-39\tokenizers\models
>       copying tokenizers\decoders\__init__.pyi -> build\lib.win-amd64-cpython-39\tokenizers\decoders
>       copying tokenizers\normalizers\__init__.pyi -> build\lib.win-amd64-cpython-39\tokenizers\normalizers
>       copying tokenizers\pre_tokenizers\__init__.pyi -> build\lib.win-amd64-cpython-39\tokenizers\pre_tokenizers
>       copying tokenizers\processors\__init__.pyi -> build\lib.win-amd64-cpython-39\tokenizers\processors
>       copying tokenizers\trainers\__init__.pyi -> build\lib.win-amd64-cpython-39\tokenizers\trainers
>       running build_ext
>       running build_rust
>       info: syncing channel updates for 'nightly-x86_64-pc-windows-msvc'
>       info: latest update on 2022-10-18, rust version 1.66.0-nightly (06f049a35 2022-10-17)
>       info: downloading component 'cargo'
>       info: downloading component 'clippy'
>       info: downloading component 'rust-docs'
>       info: downloading component 'rust-std'
>       info: downloading component 'rustc'
>       info: downloading component 'rustfmt'
>       info: installing component 'cargo'
>       info: installing component 'clippy'
>       info: installing component 'rust-docs'
>       info: retrying renaming 'C:\Users\King\.rustup\tmp\qkopjzc7kj3elau0_dir\rust-docs\share/doc/rust/html' to 'C:\Users\King\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\share/doc/rust/html'
>       info: installing component 'rust-std'
>       info: installing component 'rustc'
>       info: installing component 'rustfmt'
>       cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module -- --crate-type cdylib
>       warning: unused manifest key: target.x86_64-apple-darwin.rustflags
>           Updating crates.io index
>        Downloading crates ...
>         Downloaded autocfg v1.1.0
>         Downloaded serde v1.0.145
>         Downloaded serde_json v1.0.86
>         Downloaded rand v0.7.3
>         Downloaded itoa v1.0.4
>         Downloaded memchr v2.5.0
>         Downloaded num_cpus v1.13.1
>         Downloaded inventory-impl v0.1.11
>         Downloaded parking_lot v0.10.2
>         Downloaded parking_lot_core v0.7.2
>         Downloaded number_prefix v0.3.0
>         Downloaded indoc-impl v0.3.6
>         Downloaded serde_derive v1.0.145
>         Downloaded ryu v1.0.11
>         Downloaded version_check v0.9.4
>         Downloaded unicode-width v0.1.10
>         Downloaded syn v1.0.102
>         Downloaded ghost v0.1.6
>         Downloaded lock_api v0.3.4
>         Downloaded inventory v0.1.11
>         Downloaded indicatif v0.14.0
>         Downloaded getrandom v0.1.16
>         Downloaded textwrap v0.11.0
>         Downloaded lazy_static v1.4.0
>         Downloaded ctor v0.1.24
>         Downloaded crossbeam-utils v0.8.12
>         Downloaded num-traits v0.2.15
>         Downloaded unindent v0.1.10
>         Downloaded unicode-normalization-alignments v0.1.12
>         Downloaded pyo3cls v0.9.2
>         Downloaded pyo3 v0.9.2
>         Downloaded paste-impl v0.1.18
>         Downloaded rand_core v0.5.1
>         Downloaded strsim v0.8.0
>         Downloaded scopeguard v1.1.0
>         Downloaded vec_map v0.8.2
>         Downloaded unicode-ident v1.0.5
>         Downloaded rayon-core v1.9.3
>         Downloaded rayon v1.5.3
>         Downloaded unicode_categories v0.1.1
>         Downloaded pyo3-derive-backend v0.9.2
>         Downloaded bitflags v1.3.2
>         Downloaded cfg-if v1.0.0
>         Downloaded cfg-if v0.1.10
>         Downloaded atty v0.2.14
>         Downloaded regex v1.6.0
>         Downloaded memoffset v0.6.5
>         Downloaded indoc v0.3.6
>         Downloaded either v1.8.0
>         Downloaded crossbeam-deque v0.8.2
>         Downloaded crossbeam-channel v0.5.6
>         Downloaded clap v2.34.0
>         Downloaded regex-syntax v0.6.27
>         Downloaded winapi v0.3.9
>         Downloaded libc v0.2.135
>         Downloaded aho-corasick v0.7.19
>         Downloaded console v0.15.2
>         Downloaded crossbeam-epoch v0.9.11
>         Downloaded encode_unicode v0.3.6
>         Downloaded smallvec v1.10.0
>         Downloaded terminal_size v0.1.17
>         Downloaded proc-macro2 v1.0.47
>         Downloaded quote v1.0.21
>         Downloaded rand_chacha v0.2.2
>         Downloaded proc-macro-hack v0.5.19
>         Downloaded ppv-lite86 v0.2.16
>         Downloaded paste v0.1.18
>          Compiling proc-macro2 v1.0.47
>          Compiling quote v1.0.21
>          Compiling unicode-ident v1.0.5
>          Compiling syn v1.0.102
>          Compiling autocfg v1.1.0
>          Compiling memchr v2.5.0
>          Compiling winapi v0.3.9
>          Compiling cfg-if v1.0.0
>          Compiling serde_derive v1.0.145
>          Compiling serde v1.0.145
>          Compiling serde_json v1.0.86
>          Compiling scopeguard v1.1.0
>          Compiling crossbeam-utils v0.8.12
>          Compiling proc-macro-hack v0.5.19
>          Compiling getrandom v0.1.16
>          Compiling libc v0.2.135
>            Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-macro2-1.0.47\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"proc-macro\"" -C metadata=5b0e58d159021849 -C extra-filename=-5b0e58d159021849 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\proc-macro2-5b0e58d159021849 -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\quote-1.0.21\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"proc-macro\"" -C metadata=142419b9cedb1d17 -C extra-filename=-142419b9cedb1d17 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\quote-142419b9cedb1d17 -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name unicode_ident --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\unicode-ident-1.0.5\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off -C metadata=1afecb6243c40eff -C extra-filename=-1afecb6243c40eff --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\syn-1.0.102\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"clone-impls\"" --cfg "feature=\"default\"" --cfg "feature=\"derive\"" --cfg "feature=\"extra-traits\"" --cfg "feature=\"full\"" --cfg "feature=\"parsing\"" --cfg "feature=\"printing\"" --cfg "feature=\"proc-macro\"" --cfg "feature=\"quote\"" -C metadata=b8e33a9e5e50652b -C extra-filename=-b8e33a9e5e50652b --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\syn-b8e33a9e5e50652b -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name autocfg C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\autocfg-1.1.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off -C metadata=572016c50f479faf -C extra-filename=-572016c50f479faf --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\memchr-2.5.0\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=4ba3537c9d483825 -C extra-filename=-4ba3537c9d483825 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\memchr-4ba3537c9d483825 -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name build_script_build C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\winapi-0.3.9\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"consoleapi\"" --cfg "feature=\"errhandlingapi\"" --cfg "feature=\"fileapi\"" --cfg "feature=\"handleapi\"" --cfg "feature=\"minwinbase\"" --cfg "feature=\"minwindef\"" --cfg "feature=\"ntstatus\"" --cfg "feature=\"processenv\"" --cfg "feature=\"winbase\"" --cfg "feature=\"wincon\"" --cfg "feature=\"winerror\"" --cfg "feature=\"winnt\"" --cfg "feature=\"winuser\"" -C metadata=9ff0f262099ebbbd -C extra-filename=-9ff0f262099ebbbd --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\winapi-9ff0f262099ebbbd -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name cfg_if --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\cfg-if-1.0.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=360d109d20cdf1de -C extra-filename=-360d109d20cdf1de --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name build_script_build C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\serde_derive-1.0.145\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" -C metadata=84c8f4354cf6545a -C extra-filename=-84c8f4354cf6545a --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\serde_derive-84c8f4354cf6545a -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name build_script_build C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\serde-1.0.145\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"derive\"" --cfg "feature=\"serde_derive\"" --cfg "feature=\"std\"" -C metadata=1080dbca61ce40eb -C extra-filename=-1080dbca61ce40eb --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\serde-1080dbca61ce40eb -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\serde_json-1.0.86\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=b3ba47c415a7892f -C extra-filename=-b3ba47c415a7892f --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\serde_json-b3ba47c415a7892f -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name scopeguard C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\scopeguard-1.1.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=38978501dffc7e67 -C extra-filename=-38978501dffc7e67 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\crossbeam-utils-0.8.12\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=66c260199ab3a6b1 -C extra-filename=-66c260199ab3a6b1 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\crossbeam-utils-66c260199ab3a6b1 -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-macro-hack-0.5.19\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off -C metadata=43fb356fc25df1f2 -C extra-filename=-43fb356fc25df1f2 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\proc-macro-hack-43fb356fc25df1f2 -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\getrandom-0.1.16\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"std\"" -C metadata=c55ade8324d95812 -C extra-filename=-c55ade8324d95812 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\getrandom-c55ade8324d95812 -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name build_script_build C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\libc-0.2.135\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=e99f7f26a77660cd -C extra-filename=-e99f7f26a77660cd --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\libc-e99f7f26a77660cd -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>          Compiling unicode-width v0.1.10
>          Compiling rayon-core v1.9.3
>            Running `rustc --crate-name unicode_width C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\unicode-width-0.1.10\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"default\"" -C metadata=5f855d5352d382a0 -C extra-filename=-5f855d5352d382a0 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\rayon-core-1.9.3\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off -C metadata=e7c93bf382490bd0 -C extra-filename=-e7c93bf382490bd0 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\rayon-core-e7c93bf382490bd0 -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>          Compiling smallvec v1.10.0
>            Running `rustc --crate-name smallvec --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\smallvec-1.10.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=881054317b8f4d9c -C extra-filename=-881054317b8f4d9c --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>          Compiling ryu v1.0.11
>            Running `rustc --crate-name ryu --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\ryu-1.0.11\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off -C metadata=1d25c68059b095ff -C extra-filename=-1d25c68059b095ff --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>          Compiling memoffset v0.6.5
>          Compiling crossbeam-epoch v0.9.11
>            Running `rustc --crate-name build_script_build C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\memoffset-0.6.5\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" -C metadata=8550414a9ea9eb93 -C extra-filename=-8550414a9ea9eb93 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\memoffset-8550414a9ea9eb93 -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern autocfg=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libautocfg-572016c50f479faf.rlib --cap-lints allow`
>            Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\crossbeam-epoch-0.9.11\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"alloc\"" --cfg "feature=\"std\"" -C metadata=a5f96eb43b93974a -C extra-filename=-a5f96eb43b93974a --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\crossbeam-epoch-a5f96eb43b93974a -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern autocfg=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libautocfg-572016c50f479faf.rlib --cap-lints allow`
>          Compiling regex-syntax v0.6.27
>            Running `rustc --crate-name regex_syntax --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\regex-syntax-0.6.27\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"unicode\"" --cfg "feature=\"unicode-age\"" --cfg "feature=\"unicode-bool\"" --cfg "feature=\"unicode-case\"" --cfg "feature=\"unicode-gencat\"" --cfg "feature=\"unicode-perl\"" --cfg "feature=\"unicode-script\"" --cfg "feature=\"unicode-segment\"" -C metadata=ecd6baa27c468f0d -C extra-filename=-ecd6baa27c468f0d --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\syn-b8e33a9e5e50652b\build-script-build`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\proc-macro2-5b0e58d159021849\build-script-build`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\quote-142419b9cedb1d17\build-script-build`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\winapi-9ff0f262099ebbbd\build-script-build`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\serde_derive-84c8f4354cf6545a\build-script-build`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\memoffset-8550414a9ea9eb93\build-script-build`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\proc-macro-hack-43fb356fc25df1f2\build-script-build`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\getrandom-c55ade8324d95812\build-script-build`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\libc-e99f7f26a77660cd\build-script-build`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\serde-1080dbca61ce40eb\build-script-build`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\crossbeam-epoch-a5f96eb43b93974a\build-script-build`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\crossbeam-utils-66c260199ab3a6b1\build-script-build`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\memchr-4ba3537c9d483825\build-script-build`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\memchr-4ba3537c9d483825\build-script-build`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\rayon-core-e7c93bf382490bd0\build-script-build`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\serde_json-b3ba47c415a7892f\build-script-build`
>            Running `rustc --crate-name proc_macro2 --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-macro2-1.0.47\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"proc-macro\"" -C metadata=30944198976f606e -C extra-filename=-30944198976f606e --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern unicode_ident=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libunicode_ident-1afecb6243c40eff.rmeta --cap-lints allow --cfg use_proc_macro --cfg wrap_proc_macro --cfg proc_macro_span`
>          Compiling itoa v1.0.4
>            Running `rustc --crate-name itoa --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\itoa-1.0.4\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off -C metadata=c5e72f9360d1cff0 -C extra-filename=-c5e72f9360d1cff0 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name winapi C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\winapi-0.3.9\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"consoleapi\"" --cfg "feature=\"errhandlingapi\"" --cfg "feature=\"fileapi\"" --cfg "feature=\"handleapi\"" --cfg "feature=\"minwinbase\"" --cfg "feature=\"minwindef\"" --cfg "feature=\"ntstatus\"" --cfg "feature=\"processenv\"" --cfg "feature=\"winbase\"" --cfg "feature=\"wincon\"" --cfg "feature=\"winerror\"" --cfg "feature=\"winnt\"" --cfg "feature=\"winuser\"" -C metadata=c43802643be2e565 -C extra-filename=-c43802643be2e565 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow -l dylib=advapi32 -l dylib=cfgmgr32 -l dylib=gdi32 -l dylib=kernel32 -l dylib=msimg32 -l dylib=opengl32 -l dylib=user32 -l dylib=winspool --cfg "feature=\"vadefs\"" --cfg "feature=\"cfgmgr32\"" --cfg "feature=\"devpropdef\"" --cfg "feature=\"processthreadsapi\"" --cfg "feature=\"basetsd\"" --cfg "feature=\"winreg\"" --cfg "feature=\"ntdef\"" --cfg "feature=\"libloaderapi\"" --cfg "feature=\"wincontypes\"" --cfg "feature=\"vcruntime\"" --cfg "feature=\"excpt\"" --cfg "feature=\"wtypesbase\"" --cfg "feature=\"reason\"" --cfg "feature=\"limits\"" --cfg "feature=\"rpcndr\"" --cfg "feature=\"ktmtypes\"" --cfg "feature=\"guiddef\"" --cfg "feature=\"windef\"" --cfg "feature=\"wingdi\"" --cfg "feature=\"cfg\""`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\serde-1080dbca61ce40eb\build-script-build`
>            Running `rustc --crate-name memoffset C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\memoffset-0.6.5\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"default\"" -C metadata=d521e9d5ca1cfb74 -C extra-filename=-d521e9d5ca1cfb74 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow --cfg tuple_ty --cfg allow_clippy --cfg maybe_uninit --cfg doctests --cfg raw_ref_macros`
>            Running `rustc --crate-name proc_macro_hack --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-macro-hack-0.5.19\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debug-assertions=off -C metadata=097737d8e2c954ab -C extra-filename=-097737d8e2c954ab --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern proc_macro --cap-lints allow`
>            Running `rustc --crate-name getrandom --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\getrandom-0.1.16\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"std\"" -C metadata=6c7f93dfc6444a9c -C extra-filename=-6c7f93dfc6444a9c --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern cfg_if=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcfg_if-360d109d20cdf1de.rmeta --cap-lints allow -l advapi32`
>            Running `rustc --crate-name libc C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\libc-0.2.135\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=86a1bab4b120f6a2 -C extra-filename=-86a1bab4b120f6a2 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow --cfg freebsd11 --cfg libc_priv_mod_use --cfg libc_union --cfg libc_const_size_of --cfg libc_align --cfg libc_int128 --cfg libc_core_cvoid --cfg libc_packedN --cfg libc_cfg_target_vendor --cfg libc_non_exhaustive --cfg libc_ptr_addr_of --cfg libc_underscore_const_names --cfg libc_const_extern_fn`
>          Compiling rayon v1.5.3
>            Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\rayon-1.5.3\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off -C metadata=f6ea986aed560875 -C extra-filename=-f6ea986aed560875 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\rayon-f6ea986aed560875 -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern autocfg=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libautocfg-572016c50f479faf.rlib --cap-lints allow`
>          Compiling num-traits v0.2.15
>            Running `rustc --crate-name build_script_build C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\num-traits-0.2.15\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=286ff2921bdb81ee -C extra-filename=-286ff2921bdb81ee --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\num-traits-286ff2921bdb81ee -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern autocfg=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libautocfg-572016c50f479faf.rlib --cap-lints allow`
>            Running `rustc --crate-name crossbeam_utils --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\crossbeam-utils-0.8.12\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=3869c69ecfdeae93 -C extra-filename=-3869c69ecfdeae93 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern cfg_if=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcfg_if-360d109d20cdf1de.rmeta --cap-lints allow`
>            Running `rustc --crate-name memchr --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\memchr-2.5.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=055ae21494443b01 -C extra-filename=-055ae21494443b01 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow --cfg memchr_runtime_simd --cfg memchr_runtime_sse2 --cfg memchr_runtime_sse42 --cfg memchr_runtime_avx`
>            Running `rustc --crate-name memchr --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\memchr-2.5.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=f54a1ac40e150573 -C extra-filename=-f54a1ac40e150573 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow --cfg memchr_runtime_simd --cfg memchr_runtime_sse2 --cfg memchr_runtime_sse42 --cfg memchr_runtime_avx`
>          Compiling unindent v0.1.10
>            Running `rustc --crate-name unindent --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\unindent-0.1.10\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off -C metadata=fcf514872b9c9d8e -C extra-filename=-fcf514872b9c9d8e --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name regex_syntax --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\regex-syntax-0.6.27\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"default\"" --cfg "feature=\"unicode\"" --cfg "feature=\"unicode-age\"" --cfg "feature=\"unicode-bool\"" --cfg "feature=\"unicode-case\"" --cfg "feature=\"unicode-gencat\"" --cfg "feature=\"unicode-perl\"" --cfg "feature=\"unicode-script\"" --cfg "feature=\"unicode-segment\"" -C metadata=5a236c973584f460 -C extra-filename=-5a236c973584f460 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>          Compiling cfg-if v0.1.10
>            Running `rustc --crate-name cfg_if --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\cfg-if-0.1.10\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=f3af8d70d946c761 -C extra-filename=-f3af8d70d946c761 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>          Compiling encode_unicode v0.3.6
>            Running `rustc --crate-name encode_unicode C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\encode_unicode-0.3.6\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=3130f1c305b48f53 -C extra-filename=-3130f1c305b48f53 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>          Compiling inventory v0.1.11
>          Compiling ppv-lite86 v0.2.16
>            Running `rustc --crate-name build_script_build --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\inventory-0.1.11\build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off -C metadata=2a48a1911251848c -C extra-filename=-2a48a1911251848c --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\inventory-2a48a1911251848c -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name ppv_lite86 --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\ppv-lite86-0.2.16\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"simd\"" --cfg "feature=\"std\"" -C metadata=94c9eb93ee689d3b -C extra-filename=-94c9eb93ee689d3b --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>          Compiling rand_core v0.5.1
>            Running `rustc --crate-name rand_core --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\rand_core-0.5.1\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"alloc\"" --cfg "feature=\"getrandom\"" --cfg "feature=\"std\"" -C metadata=06e7d83191242dc6 -C extra-filename=-06e7d83191242dc6 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern getrandom=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libgetrandom-6c7f93dfc6444a9c.rmeta --cap-lints allow`
>          Compiling version_check v0.9.4
>            Running `rustc --crate-name version_check C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\version_check-0.9.4\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off -C metadata=0584bbe3f7491e0d -C extra-filename=-0584bbe3f7491e0d --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name quote --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\quote-1.0.21\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"proc-macro\"" -C metadata=dabbd39e3909793b -C extra-filename=-dabbd39e3909793b --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern proc_macro2=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libproc_macro2-30944198976f606e.rmeta --cap-lints allow`
>          Compiling num_cpus v1.13.1
>            Running `rustc --crate-name num_cpus C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\num_cpus-1.13.1\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=31ad845e15e2b15d -C extra-filename=-31ad845e15e2b15d --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>          Compiling lazy_static v1.4.0
>            Running `rustc --crate-name lazy_static C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\lazy_static-1.4.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=431f071606c587f4 -C extra-filename=-431f071606c587f4 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>            Running `rustc --crate-name crossbeam_epoch --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\crossbeam-epoch-0.9.11\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"alloc\"" --cfg "feature=\"std\"" -C metadata=4897f65d76bcc39c -C extra-filename=-4897f65d76bcc39c --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern cfg_if=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcfg_if-360d109d20cdf1de.rmeta --extern crossbeam_utils=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcrossbeam_utils-3869c69ecfdeae93.rmeta --extern memoffset=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libmemoffset-d521e9d5ca1cfb74.rmeta --extern scopeguard=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libscopeguard-38978501dffc7e67.rmeta --cap-lints allow`
>          Compiling crossbeam-channel v0.5.6
>            Running `rustc --crate-name crossbeam_channel --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\crossbeam-channel-0.5.6\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"crossbeam-utils\"" --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=658fbd4d6c655a6f -C extra-filename=-658fbd4d6c655a6f --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern cfg_if=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcfg_if-360d109d20cdf1de.rmeta --extern crossbeam_utils=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcrossbeam_utils-3869c69ecfdeae93.rmeta --cap-lints allow`
>            Running `rustc --crate-name syn --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\syn-1.0.102\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"clone-impls\"" --cfg "feature=\"default\"" --cfg "feature=\"derive\"" --cfg "feature=\"extra-traits\"" --cfg "feature=\"full\"" --cfg "feature=\"parsing\"" --cfg "feature=\"printing\"" --cfg "feature=\"proc-macro\"" --cfg "feature=\"quote\"" -C metadata=ed20b2b04ea8255e -C extra-filename=-ed20b2b04ea8255e --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern proc_macro2=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libproc_macro2-30944198976f606e.rmeta --extern quote=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libquote-dabbd39e3909793b.rmeta --extern unicode_ident=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libunicode_ident-1afecb6243c40eff.rmeta --cap-lints allow`
>          Compiling aho-corasick v0.7.19
>            Running `rustc --crate-name aho_corasick --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\aho-corasick-0.7.19\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=f1529ca83e28da6f -C extra-filename=-f1529ca83e28da6f --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern memchr=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libmemchr-055ae21494443b01.rmeta --cap-lints allow`
>          Compiling rand_chacha v0.2.2
>            Running `rustc --crate-name aho_corasick --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\aho-corasick-0.7.19\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=dfcbc49062ee55e6 -C extra-filename=-dfcbc49062ee55e6 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern memchr=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libmemchr-f54a1ac40e150573.rmeta --cap-lints allow`
>            Running `rustc --crate-name rand_chacha --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\rand_chacha-0.2.2\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"std\"" -C metadata=42edb5e8f22d28e3 -C extra-filename=-42edb5e8f22d28e3 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern ppv_lite86=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libppv_lite86-94c9eb93ee689d3b.rmeta --extern rand_core=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\librand_core-06e7d83191242dc6.rmeta --cap-lints allow`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\inventory-2a48a1911251848c\build-script-build`
>          Compiling crossbeam-deque v0.8.2
>            Running `rustc --crate-name crossbeam_deque --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\crossbeam-deque-0.8.2\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"crossbeam-epoch\"" --cfg "feature=\"crossbeam-utils\"" --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=b75d3846468b682e -C extra-filename=-b75d3846468b682e --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern cfg_if=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcfg_if-360d109d20cdf1de.rmeta --extern crossbeam_epoch=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcrossbeam_epoch-4897f65d76bcc39c.rmeta --extern crossbeam_utils=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcrossbeam_utils-3869c69ecfdeae93.rmeta --cap-lints allow`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\num-traits-286ff2921bdb81ee\build-script-build`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\rayon-f6ea986aed560875\build-script-build`
>            Running `C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\build\serde_json-b3ba47c415a7892f\build-script-build`
>          Compiling textwrap v0.11.0
>            Running `rustc --crate-name rayon_core --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\rayon-core-1.9.3\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=0dc7012c06ddf19e -C extra-filename=-0dc7012c06ddf19e --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern crossbeam_channel=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcrossbeam_channel-658fbd4d6c655a6f.rmeta --extern crossbeam_deque=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcrossbeam_deque-b75d3846468b682e.rmeta --extern crossbeam_utils=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libcrossbeam_utils-3869c69ecfdeae93.rmeta --extern num_cpus=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libnum_cpus-31ad845e15e2b15d.rmeta --cap-lints allow`
>            Running `rustc --crate-name textwrap C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\textwrap-0.11.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=65eec7bbb3cdf0f8 -C extra-filename=-65eec7bbb3cdf0f8 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern unicode_width=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libunicode_width-5f855d5352d382a0.rmeta --cap-lints allow`
>          Compiling paste-impl v0.1.18
>            Running `rustc --crate-name paste_impl --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\paste-impl-0.1.18\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debug-assertions=off -C metadata=d5e32c087e023b55 -C extra-filename=-d5e32c087e023b55 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern proc_macro_hack=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\proc_macro_hack-097737d8e2c954ab.dll --extern proc_macro --cap-lints allow`
>          Compiling lock_api v0.3.4
>          Compiling strsim v0.8.0
>            Running `rustc --crate-name lock_api --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\lock_api-0.3.4\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"nightly\"" -C metadata=2fe17ee22c58d967 -C extra-filename=-2fe17ee22c58d967 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern scopeguard=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libscopeguard-38978501dffc7e67.rmeta --cap-lints allow`
>            Running `rustc --crate-name strsim C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\strsim-0.8.0\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C metadata=8fb9b7eda8bf3ac7 -C extra-filename=-8fb9b7eda8bf3ac7 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --cap-lints allow`
>       error[E0557]: feature has been removed
>         --> C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\lock_api-0.3.4\src\lib.rs:91:42
>          |
>       91 | #![cfg_attr(feature = "nightly", feature(const_fn))]
>          |                                          ^^^^^^^^ feature has been removed
>          |
>          = note: split into finer-grained feature gates
> 
>       For more information about this error, try `rustc --explain E0557`.
>       error: could not compile `lock_api` due to previous error
> 
>       Caused by:
>         process didn't exit successfully: `rustc --crate-name lock_api --edition=2018 C:\Users\King\.cargo\registry\src\github.com-1ecc6299db9ec823\lock_api-0.3.4\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"nightly\"" -C metadata=2fe17ee22c58d967 -C extra-filename=-2fe17ee22c58d967 --out-dir C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps -L dependency=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps --extern scopeguard=C:\Users\King\AppData\Local\Temp\pip-install-b7_bo2sd\tokenizers_1b6098a3d736487c996866dfd304f880\target\release\deps\libscopeguard-38978501dffc7e67.rmeta --cap-lints allow` (exit code: 1)
>       warning: build failed, waiting for other jobs to finish...
>       error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module -- --crate-type cdylib` failed with code 101
>       [end of output]
> 
>   note: This error originates from a subprocess, and is likely not a problem with pip.
>   ERROR: Failed building wheel for tokenizers
> Failed to build tokenizers
> ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects
> 
> C:\punctuation-restoration>
> ```

I have the same python version and the exactly same question. I am running on Mac 2. I tried to lower the version of transformers and rustc. Both did not work out.

could you try to update to the latest tokenizers version? We published a release yesterday! 

> could you try to update to the latest tokenizers version? We published a release yesterday!

Wow. It is working properly now!!! Thanks for the support. 




> could you try to update to the latest tokenizers version? We published a release yesterday!

@ArthurZucker   that is ok, when transformers==4.37.2,tokenizers is fixed!!

> > 您可以尝试更新到最新的分词器版本吗？我们昨天发布了一个版本！
> 
> @ArthurZucker 没关系，当 Transformers==4.37.2 时，Tokenizers 就被修复了！

Thanks! It looks good to me!

I am facing error "ModuleNotFoundError: No module named 'abouchet' " so i tried installing  pip install transformers==4.2 
, faces "Failed building wheel for tokenizers ". i can not find solution here.


> could you try to update to the latest tokenizers version? We published a release yesterday!

Sorry for the stupid question but how do you update tokenizers when installing transformers? I did a pip install tokenizers first and that went without issues but pip install transformers still fails. Fails hard on any of the specific transformers (transformer[tf-cpu] etc from the huggingface docs

Mybe python version is not match with tokenizers library.

Yes. Older version of tokenizers were not built for newer version of python. 
The best here @shividwivedi is to build from source

So basically it's transformers fault for not updating tokenizers?! pip install transformer worked at the end btw - pip install transformer[tf-cpu] did not

No it's not `transformers`'s fault, tokenizers' build are different from `transformers`, and depending on the version you want we were not building the wheels for python > 3.9 for example 

I had the same issues on Windows 11 (64 Bit)
Solved by installing transformers==4.37.2 

God blesses!!!

Regards

I was getting the same error while installing The transformers Library From hugginface, Specifically while installing flax and tensor flow

i Found a Solution To my problem, Leaving it here in case Anyone Else gets Stuck on this.

pip install 'transformers[tf-cpu]'

pip install 'transformers[flax]'  (these commands are given on hugginface transformers installation page)

i tried multiple things but nothing i saw on the internet Worked.

then out of Frustration i installed Flax and TF(Tensor-Flow) like normal commands inside a virtual python Env,

pip install tf

pip install flax

and they Worked.

maybe this will help someone.