Cannot install Transformers version >2.3.0 with pip on CentOS
Author: JohnGiorgi
Pull request: no
Labels: name: Core: Tokenization

name: Installation

# 🐛 Bug

I cannot install `pip install transformers` for a release newer than `2.3.0`. The install errors out when trying to install `tokenizers`. This is similar to [another issue](https://github.com/huggingface/transformers/issues/2831), except I have a Rust Compiler in my environment so I do not see: `"error: can not find Rust Compiler"`.

## Information

Model I am using (Bert, XLNet ...):

N/A

Language I am using the model on (English, Chinese ...):

N/A

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
pip install transformers
```

which leads to the following error:

```
Building wheels for collected packages: tokenizers
  Building wheel for tokenizers (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/johnmg/t2t/bin/python /home/johnmg/t2t/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpcvv0fpj6
       cwd: /tmp/pip-install-d2wcoxbe/tokenizers
  Complete output (221 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib
  creating build/lib/tokenizers
  copying tokenizers/__init__.py -> build/lib/tokenizers
  creating build/lib/tokenizers/models
  copying tokenizers/models/__init__.py -> build/lib/tokenizers/models
  creating build/lib/tokenizers/decoders
  copying tokenizers/decoders/__init__.py -> build/lib/tokenizers/decoders
  creating build/lib/tokenizers/normalizers
  copying tokenizers/normalizers/__init__.py -> build/lib/tokenizers/normalizers
  creating build/lib/tokenizers/pre_tokenizers
  copying tokenizers/pre_tokenizers/__init__.py -> build/lib/tokenizers/pre_tokenizers
  creating build/lib/tokenizers/processors
  copying tokenizers/processors/__init__.py -> build/lib/tokenizers/processors
  creating build/lib/tokenizers/trainers
  copying tokenizers/trainers/__init__.py -> build/lib/tokenizers/trainers
  creating build/lib/tokenizers/implementations
  copying tokenizers/implementations/sentencepiece_bpe.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/char_level_bpe.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/byte_level_bpe.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/bert_wordpiece.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/base_tokenizer.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/__init__.py -> build/lib/tokenizers/implementations
  copying tokenizers/__init__.pyi -> build/lib/tokenizers
  copying tokenizers/models/__init__.pyi -> build/lib/tokenizers/models
  copying tokenizers/decoders/__init__.pyi -> build/lib/tokenizers/decoders
  copying tokenizers/normalizers/__init__.pyi -> build/lib/tokenizers/normalizers
  copying tokenizers/pre_tokenizers/__init__.pyi -> build/lib/tokenizers/pre_tokenizers
  copying tokenizers/processors/__init__.pyi -> build/lib/tokenizers/processors
  copying tokenizers/trainers/__init__.pyi -> build/lib/tokenizers/trainers
  running build_ext
  running build_rust
      Updating crates.io index
  warning: unused manifest key: target.x86_64-apple-darwin.rustflags
     Compiling proc-macro2 v1.0.8
     Compiling unicode-xid v0.2.0
     Compiling syn v1.0.15
     Compiling libc v0.2.67
     Compiling autocfg v1.0.0
     Compiling lazy_static v1.4.0
     Compiling cfg-if v0.1.10
     Compiling semver-parser v0.7.0
     Compiling memchr v2.3.3
     Compiling serde v1.0.104
     Compiling maybe-uninit v2.0.0
     Compiling ryu v1.0.2
     Compiling regex-syntax v0.6.14
     Compiling getrandom v0.1.14
     Compiling scopeguard v1.1.0
     Compiling unicode-width v0.1.7
     Compiling itoa v0.4.5
     Compiling bitflags v1.2.1
       Running `rustc --crate-name unicode_xid /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-xid-0.2.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' -C metadata=d0c8361b0afb9c55 -C extra-filename=-d0c8361b0afb9c55 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --edition=2018 --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-1.0.8/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=97f59661e87a2bff -C extra-filename=-97f59661e87a2bff --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/proc-macro2-97f59661e87a2bff -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --edition=2018 --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/syn-1.0.15/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="clone-impls"' --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="extra-traits"' --cfg 'feature="full"' --cfg 'feature="parsing"' --cfg 'feature="printing"' --cfg 'feature="proc-macro"' --cfg 'feature="quote"' --cfg 'feature="visit"' -C metadata=be4245bf41be9154 -C extra-filename=-be4245bf41be9154 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/syn-be4245bf41be9154 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.67/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=4aa17b314a9f9392 -C extra-filename=-4aa17b314a9f9392 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/libc-4aa17b314a9f9392 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name autocfg /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/autocfg-1.0.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=0b99959d54eb5a43 -C extra-filename=-0b99959d54eb5a43 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name lazy_static /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/lazy_static-1.4.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=7fe90463f0542b89 -C extra-filename=-7fe90463f0542b89 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name semver_parser /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/semver-parser-0.7.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=ce71380f50d590b6 -C extra-filename=-ce71380f50d590b6 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --edition=2018 --crate-name cfg_if /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/cfg-if-0.1.10/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=1b04fa8f4baea64e -C extra-filename=-1b04fa8f4baea64e --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/memchr-2.3.3/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' --cfg 'feature="use_std"' -C metadata=682166ccfd58c578 -C extra-filename=-682166ccfd58c578 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/memchr-682166ccfd58c578 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.104/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="serde_derive"' --cfg 'feature="std"' -C metadata=e3191056f1858817 -C extra-filename=-e3191056f1858817 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/serde-e3191056f1858817 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/maybe-uninit-2.0.0/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=99bcd9a60d46382c -C extra-filename=-99bcd9a60d46382c --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/maybe-uninit-99bcd9a60d46382c -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ryu-1.0.2/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=e704a6b7a71f3d7a -C extra-filename=-e704a6b7a71f3d7a --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/ryu-e704a6b7a71f3d7a -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name regex_syntax /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/regex-syntax-0.6.14/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="unicode"' --cfg 'feature="unicode-age"' --cfg 'feature="unicode-bool"' --cfg 'feature="unicode-case"' --cfg 'feature="unicode-gencat"' --cfg 'feature="unicode-perl"' --cfg 'feature="unicode-script"' --cfg 'feature="unicode-segment"' -C metadata=feb44197369905d4 -C extra-filename=-feb44197369905d4 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --edition=2018 --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/getrandom-0.1.14/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="std"' -C metadata=c2394b8b43d330b2 -C extra-filename=-c2394b8b43d330b2 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/getrandom-c2394b8b43d330b2 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name scopeguard /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/scopeguard-1.1.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=8a63ed9d96488c18 -C extra-filename=-8a63ed9d96488c18 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name unicode_width /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-width-0.1.7/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' -C metadata=397b6227577b65ae -C extra-filename=-397b6227577b65ae --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name itoa /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/itoa-0.4.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=b9ca519a13df71bf -C extra-filename=-b9ca519a13df71bf --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
     Compiling ppv-lite86 v0.2.6
     Compiling rayon-core v1.7.0
     Compiling unindent v0.1.5
     Compiling version_check v0.9.1
     Compiling strsim v0.8.0
     Compiling vec_map v0.8.1
     Compiling either v1.5.3
     Compiling number_prefix v0.3.0
     Compiling smallvec v1.2.0
     Compiling ansi_term v0.11.0
     Compiling unicode_categories v0.1.1
     Compiling spin v0.5.2
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/bitflags-1.2.1/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' -C metadata=74dee2c088f4fdf7 -C extra-filename=-74dee2c088f4fdf7 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/bitflags-74dee2c088f4fdf7 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --edition=2018 --crate-name ppv_lite86 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="simd"' --cfg 'feature="std"' -C metadata=a66cdba604de2e91 -C extra-filename=-a66cdba604de2e91 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --edition=2018 --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.7.0/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=01d02e6ac28bd2a9 -C extra-filename=-01d02e6ac28bd2a9 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/rayon-core-01d02e6ac28bd2a9 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name version_check /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/version_check-0.9.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=cb4ea0451d56bc0d -C extra-filename=-cb4ea0451d56bc0d --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --edition=2018 --crate-name unindent /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/unindent-0.1.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=54ee88ab0038e61b -C extra-filename=-54ee88ab0038e61b --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name strsim /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/strsim-0.8.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=97f24fc34d9d28cd -C extra-filename=-97f24fc34d9d28cd --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name vec_map /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/vec_map-0.8.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=135a62f8a977656a -C extra-filename=-135a62f8a977656a --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name either /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/either-1.5.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=6fcca30f288e7f70 -C extra-filename=-6fcca30f288e7f70 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name number_prefix /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/number_prefix-0.3.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=c617a5b231fd33f2 -C extra-filename=-c617a5b231fd33f2 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --edition=2018 --crate-name smallvec /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/smallvec-1.2.0/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=5a1b93c8a07d924a -C extra-filename=-5a1b93c8a07d924a --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name ansi_term /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ansi_term-0.11.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=f9c541f6d3ce7af3 -C extra-filename=-f9c541f6d3ce7af3 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name unicode_categories /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode_categories-0.1.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=3376acf28b9791fe -C extra-filename=-3376acf28b9791fe --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name spin /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/spin-0.5.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=6c5053f023d06140 -C extra-filename=-6c5053f023d06140 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
     Compiling thread_local v1.0.1
       Running `rustc --crate-name thread_local /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/thread_local-1.0.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=7cdd46b26d4f9805 -C extra-filename=-7cdd46b26d4f9805 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --cap-lints allow`
     Compiling textwrap v0.11.0
       Running `rustc --crate-name textwrap /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/textwrap-0.11.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=0f312c8508ee8e9d -C extra-filename=-0f312c8508ee8e9d --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern unicode_width=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_width-397b6227577b65ae.rmeta --cap-lints allow`
     Compiling semver v0.9.0
       Running `rustc --crate-name semver /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/semver-0.9.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' -C metadata=21f81775743ad422 -C extra-filename=-21f81775743ad422 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern semver_parser=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsemver_parser-ce71380f50d590b6.rmeta --cap-lints allow`
     Compiling unicode-normalization-alignments v0.1.12
       Running `rustc --crate-name unicode_normalization_alignments /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-normalization-alignments-0.1.12/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=2f95bcef5d770e35 -C extra-filename=-2f95bcef5d770e35 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern smallvec=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsmallvec-5a1b93c8a07d924a.rmeta --cap-lints allow`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/rayon-core-01d02e6ac28bd2a9/build-script-build`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/memchr-682166ccfd58c578/build-script-build`
       Running `rustc --crate-name memchr /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/memchr-2.3.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' --cfg 'feature="use_std"' -C metadata=916f9f60d041f29f -C extra-filename=-916f9f60d041f29f --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg memchr_runtime_simd --cfg memchr_runtime_sse2 --cfg memchr_runtime_sse42 --cfg memchr_runtime_avx`
     Compiling rustc_version v0.2.3
       Running `rustc --crate-name rustc_version /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rustc_version-0.2.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=63af08b6d5f0b1a9 -C extra-filename=-63af08b6d5f0b1a9 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern semver=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsemver-21f81775743ad422.rmeta --cap-lints allow`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/getrandom-c2394b8b43d330b2/build-script-build`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/bitflags-74dee2c088f4fdf7/build-script-build`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/maybe-uninit-99bcd9a60d46382c/build-script-build`
     Compiling crossbeam-utils v0.7.2
     Compiling crossbeam-epoch v0.8.2
     Compiling num-traits v0.2.11
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-utils-0.7.2/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=5b24f08aed575110 -C extra-filename=-5b24f08aed575110 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/crossbeam-utils-5b24f08aed575110 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern autocfg=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libautocfg-0b99959d54eb5a43.rlib --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=515968de6557b6b7 -C extra-filename=-515968de6557b6b7 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/crossbeam-epoch-515968de6557b6b7 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern autocfg=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libautocfg-0b99959d54eb5a43.rlib --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/num-traits-0.2.11/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=618c6828911959ab -C extra-filename=-618c6828911959ab --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/num-traits-618c6828911959ab -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern autocfg=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libautocfg-0b99959d54eb5a43.rlib --cap-lints allow`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/ryu-e704a6b7a71f3d7a/build-script-build`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/serde-e3191056f1858817/build-script-build`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/libc-4aa17b314a9f9392/build-script-build`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/syn-be4245bf41be9154/build-script-build`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/proc-macro2-97f59661e87a2bff/build-script-build`
       Running `rustc --crate-name bitflags /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/bitflags-1.2.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' -C metadata=95683541c46a6653 -C extra-filename=-95683541c46a6653 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg bitflags_const_fn`
       Running `rustc --crate-name maybe_uninit /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/maybe-uninit-2.0.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=169701ffcef7a104 -C extra-filename=-169701ffcef7a104 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg derive_copy --cfg repr_transparent --cfg native_uninit`
     Compiling c2-chacha v0.2.3
       Running `rustc --edition=2018 --crate-name c2_chacha /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/c2-chacha-0.2.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="simd"' --cfg 'feature="std"' -C metadata=db6d7fc899faf453 -C extra-filename=-db6d7fc899faf453 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern ppv_lite86=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libppv_lite86-a66cdba604de2e91.rmeta --cap-lints allow`
       Running `rustc --crate-name ryu /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ryu-1.0.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=81c16096c65f1d25 -C extra-filename=-81c16096c65f1d25 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg integer128 --cfg must_use_return --cfg maybe_uninit`
       Running `rustc --edition=2018 --crate-name proc_macro2 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-1.0.8/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=ce62abe820ec95ab -C extra-filename=-ce62abe820ec95ab --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern unicode_xid=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_xid-d0c8361b0afb9c55.rmeta --cap-lints allow --cfg use_proc_macro --cfg wrap_proc_macro`
       Running `rustc --crate-name libc /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.67/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=5d9b252ed56b1945 -C extra-filename=-5d9b252ed56b1945 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg freebsd11 --cfg libc_priv_mod_use --cfg libc_union --cfg libc_const_size_of --cfg libc_align --cfg libc_core_cvoid --cfg libc_packedN`
     Compiling aho-corasick v0.7.8
       Running `rustc --crate-name aho_corasick /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/aho-corasick-0.7.8/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=5f8df0f6460a66c2 -C extra-filename=-5f8df0f6460a66c2 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern memchr=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libmemchr-916f9f60d041f29f.rmeta --cap-lints allow`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/num-traits-618c6828911959ab/build-script-build`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/crossbeam-utils-5b24f08aed575110/build-script-build`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/crossbeam-epoch-515968de6557b6b7/build-script-build`
     Compiling memoffset v0.5.3
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/memoffset-0.5.3/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=8ebd67a7766256e7 -C extra-filename=-8ebd67a7766256e7 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/memoffset-8ebd67a7766256e7 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern rustc_version=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librustc_version-63af08b6d5f0b1a9.rlib --cap-lints allow`
     Compiling quote v1.0.2
       Running `rustc --edition=2018 --crate-name quote /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/quote-1.0.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=5dd3b63b3c37ba50 -C extra-filename=-5dd3b63b3c37ba50 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rmeta --cap-lints allow`
       Running `rustc --crate-name num_traits /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/num-traits-0.2.11/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=682c148c69950086 -C extra-filename=-682c148c69950086 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg has_i128`
     Compiling num_cpus v1.12.0
     Compiling termios v0.3.1
     Compiling clicolors-control v1.0.1
     Compiling atty v0.2.14
       Running `rustc --edition=2018 --crate-name getrandom /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/getrandom-0.1.14/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="std"' -C metadata=3bcce62cba29d0a1 -C extra-filename=-3bcce62cba29d0a1 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern cfg_if=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcfg_if-1b04fa8f4baea64e.rmeta --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --cap-lints allow`
       Running `rustc --crate-name num_cpus /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/num_cpus-1.12.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=5375d111fec819b4 -C extra-filename=-5375d111fec819b4 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --cap-lints allow`
       Running `rustc --crate-name termios /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/termios-0.3.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=d64097ba20dddbc5 -C extra-filename=-d64097ba20dddbc5 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --cap-lints allow`
       Running `rustc --crate-name clicolors_control /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/clicolors-control-1.0.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="terminal_autoconfig"' -C metadata=f95aedfd36305d68 -C extra-filename=-f95aedfd36305d68 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --cap-lints allow`
       Running `rustc --crate-name atty /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/atty-0.2.14/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=61c34de20facc8fb -C extra-filename=-61c34de20facc8fb --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --cap-lints allow`
       Running `rustc --edition=2018 --crate-name syn /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/syn-1.0.15/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="clone-impls"' --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="extra-traits"' --cfg 'feature="full"' --cfg 'feature="parsing"' --cfg 'feature="printing"' --cfg 'feature="proc-macro"' --cfg 'feature="quote"' --cfg 'feature="visit"' -C metadata=fb7a652ed3ecc931 -C extra-filename=-fb7a652ed3ecc931 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rmeta --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rmeta --extern unicode_xid=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_xid-d0c8361b0afb9c55.rmeta --cap-lints allow --cfg syn_disable_nightly_tests`
     Compiling clap v2.33.0
       Running `rustc --crate-name clap /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-2.33.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="ansi_term"' --cfg 'feature="atty"' --cfg 'feature="color"' --cfg 'feature="default"' --cfg 'feature="strsim"' --cfg 'feature="suggestions"' --cfg 'feature="vec_map"' -C metadata=4d1679758f5cc3c5 -C extra-filename=-4d1679758f5cc3c5 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern ansi_term=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libansi_term-f9c541f6d3ce7af3.rmeta --extern atty=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libatty-61c34de20facc8fb.rmeta --extern bitflags=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libbitflags-95683541c46a6653.rmeta --extern strsim=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libstrsim-97f24fc34d9d28cd.rmeta --extern textwrap=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libtextwrap-0f312c8508ee8e9d.rmeta --extern unicode_width=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_width-397b6227577b65ae.rmeta --extern vec_map=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libvec_map-135a62f8a977656a.rmeta --cap-lints allow`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/memoffset-8ebd67a7766256e7/build-script-build`
     Compiling rand_core v0.5.1
       Running `rustc --edition=2018 --crate-name rand_core /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rand_core-0.5.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="alloc"' --cfg 'feature="getrandom"' --cfg 'feature="std"' -C metadata=4adb25904fdd70df -C extra-filename=-4adb25904fdd70df --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern getrandom=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libgetrandom-3bcce62cba29d0a1.rmeta --cap-lints allow`
       Running `rustc --crate-name memoffset /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/memoffset-0.5.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=ee652fbed0600815 -C extra-filename=-ee652fbed0600815 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg memoffset_maybe_uninit --cfg memoffset_doctests`
       Running `rustc --crate-name crossbeam_utils /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-utils-0.7.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=71c95db82240db48 -C extra-filename=-71c95db82240db48 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern cfg_if=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcfg_if-1b04fa8f4baea64e.rmeta --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --cap-lints allow --cfg has_min_const_fn --cfg has_atomic_u8 --cfg has_atomic_u16 --cfg has_atomic_u32 --cfg has_atomic_u64`
     Compiling rand_chacha v0.2.1
       Running `rustc --edition=2018 --crate-name rand_chacha /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rand_chacha-0.2.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="std"' -C metadata=164a44df65235912 -C extra-filename=-164a44df65235912 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern c2_chacha=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libc2_chacha-db6d7fc899faf453.rmeta --extern rand_core=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librand_core-4adb25904fdd70df.rmeta --cap-lints allow`
     Compiling rand v0.7.3
       Running `rustc --edition=2018 --crate-name rand /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rand-0.7.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="alloc"' --cfg 'feature="default"' --cfg 'feature="getrandom"' --cfg 'feature="getrandom_package"' --cfg 'feature="libc"' --cfg 'feature="std"' -C metadata=102d035e4ca6c699 -C extra-filename=-102d035e4ca6c699 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern getrandom_package=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libgetrandom-3bcce62cba29d0a1.rmeta --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --extern rand_chacha=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librand_chacha-164a44df65235912.rmeta --extern rand_core=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librand_core-4adb25904fdd70df.rmeta --cap-lints allow`
     Compiling crossbeam-queue v0.2.1
       Running `rustc --crate-name crossbeam_epoch /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=4cd0c2190306aa4a -C extra-filename=-4cd0c2190306aa4a --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern cfg_if=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcfg_if-1b04fa8f4baea64e.rmeta --extern crossbeam_utils=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_utils-71c95db82240db48.rmeta --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --extern maybe_uninit=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libmaybe_uninit-169701ffcef7a104.rmeta --extern memoffset=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libmemoffset-ee652fbed0600815.rmeta --extern scopeguard=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libscopeguard-8a63ed9d96488c18.rmeta --cap-lints allow --cfg has_min_const_fn`
       Running `rustc --crate-name crossbeam_queue /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-queue-0.2.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=e5afea70501509a9 -C extra-filename=-e5afea70501509a9 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern cfg_if=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcfg_if-1b04fa8f4baea64e.rmeta --extern crossbeam_utils=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_utils-71c95db82240db48.rmeta --cap-lints allow`
     Compiling regex v1.3.4
       Running `rustc --crate-name regex /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/regex-1.3.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="aho-corasick"' --cfg 'feature="default"' --cfg 'feature="memchr"' --cfg 'feature="perf"' --cfg 'feature="perf-cache"' --cfg 'feature="perf-dfa"' --cfg 'feature="perf-inline"' --cfg 'feature="perf-literal"' --cfg 'feature="std"' --cfg 'feature="thread_local"' --cfg 'feature="unicode"' --cfg 'feature="unicode-age"' --cfg 'feature="unicode-bool"' --cfg 'feature="unicode-case"' --cfg 'feature="unicode-gencat"' --cfg 'feature="unicode-perl"' --cfg 'feature="unicode-script"' --cfg 'feature="unicode-segment"' -C metadata=40c5630aef8afe3e -C extra-filename=-40c5630aef8afe3e --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern aho_corasick=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libaho_corasick-5f8df0f6460a66c2.rmeta --extern memchr=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libmemchr-916f9f60d041f29f.rmeta --extern regex_syntax=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libregex_syntax-feb44197369905d4.rmeta --extern thread_local=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libthread_local-7cdd46b26d4f9805.rmeta --cap-lints allow`
     Compiling crossbeam-deque v0.7.3
       Running `rustc --crate-name crossbeam_deque /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-deque-0.7.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=6fff16ed40375025 -C extra-filename=-6fff16ed40375025 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern crossbeam_epoch=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_epoch-4cd0c2190306aa4a.rmeta --extern crossbeam_utils=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_utils-71c95db82240db48.rmeta --extern maybe_uninit=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libmaybe_uninit-169701ffcef7a104.rmeta --cap-lints allow`
       Running `rustc --edition=2018 --crate-name rayon_core /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.7.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=5e812e4a0a947026 -C extra-filename=-5e812e4a0a947026 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern crossbeam_deque=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_deque-6fff16ed40375025.rmeta --extern crossbeam_queue=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_queue-e5afea70501509a9.rmeta --extern crossbeam_utils=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_utils-71c95db82240db48.rmeta --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --extern num_cpus=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libnum_cpus-5375d111fec819b4.rmeta --cap-lints allow`
     Compiling rayon v1.3.0
       Running `rustc --edition=2018 --crate-name rayon /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.3.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=35b9846c719edc99 -C extra-filename=-35b9846c719edc99 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern crossbeam_deque=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_deque-6fff16ed40375025.rmeta --extern either=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libeither-6fcca30f288e7f70.rmeta --extern rayon_core=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librayon_core-5e812e4a0a947026.rmeta --cap-lints allow`
     Compiling console v0.9.2
       Running `rustc --edition=2018 --crate-name console /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/console-0.9.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="unicode-width"' -C metadata=a0e410a25b05d297 -C extra-filename=-a0e410a25b05d297 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern clicolors_control=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libclicolors_control-f95aedfd36305d68.rmeta --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --extern regex=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libregex-40c5630aef8afe3e.rmeta --extern termios=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libtermios-d64097ba20dddbc5.rmeta --extern unicode_width=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_width-397b6227577b65ae.rmeta --cap-lints allow`
     Compiling indicatif v0.14.0
       Running `rustc --edition=2018 --crate-name indicatif /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/indicatif-0.14.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' -C metadata=af81cb79d58cbea4 -C extra-filename=-af81cb79d58cbea4 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern console=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libconsole-a0e410a25b05d297.rmeta --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --extern number_prefix=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libnumber_prefix-c617a5b231fd33f2.rmeta --extern regex=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libregex-40c5630aef8afe3e.rmeta --cap-lints allow`
     Compiling pyo3-derive-backend v0.8.5
       Running `rustc --edition=2018 --crate-name pyo3_derive_backend /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-derive-backend-0.8.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=4e1d8a522a8e0abc -C extra-filename=-4e1d8a522a8e0abc --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rmeta --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rmeta --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rmeta --cap-lints allow`
     Compiling serde_derive v1.0.104
     Compiling proc-macro-hack v0.5.11
     Compiling ctor v0.1.12
     Compiling ghost v0.1.1
     Compiling inventory-impl v0.1.5
     Compiling pyo3cls v0.8.5
       Running `rustc --crate-name serde_derive /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_derive-1.0.104/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 --cfg 'feature="default"' -C metadata=c97a5ca23329a0e7 -C extra-filename=-c97a5ca23329a0e7 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
       Running `rustc --edition=2018 --crate-name proc_macro_hack /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro-hack-0.5.11/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=6c301fa525410f51 -C extra-filename=-6c301fa525410f51 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
       Running `rustc --edition=2018 --crate-name ctor /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ctor-0.1.12/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=6d15aecbd8ecf9a9 -C extra-filename=-6d15aecbd8ecf9a9 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
       Running `rustc --edition=2018 --crate-name ghost /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ghost-0.1.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=a7fa8d8cb581322e -C extra-filename=-a7fa8d8cb581322e --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
       Running `rustc --edition=2018 --crate-name inventory_impl /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/inventory-impl-0.1.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=07295296bec98a10 -C extra-filename=-07295296bec98a10 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
       Running `rustc --edition=2018 --crate-name pyo3cls /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3cls-0.8.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=4e128ea32e108d4e -C extra-filename=-4e128ea32e108d4e --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern pyo3_derive_backend=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libpyo3_derive_backend-4e1d8a522a8e0abc.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
     Compiling paste-impl v0.1.7
     Compiling indoc-impl v0.3.4
       Running `rustc --edition=2018 --crate-name paste_impl /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/paste-impl-0.1.7/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=0100b2f59cf859eb -C extra-filename=-0100b2f59cf859eb --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro_hack=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro_hack-6c301fa525410f51.so --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
       Running `rustc --edition=2018 --crate-name indoc_impl /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/indoc-impl-0.3.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=b6ef86cd971e397d -C extra-filename=-b6ef86cd971e397d --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro_hack=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro_hack-6c301fa525410f51.so --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --extern unindent=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunindent-54ee88ab0038e61b.rlib --cap-lints allow`
     Compiling inventory v0.1.5
       Running `rustc --edition=2018 --crate-name inventory /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/inventory-0.1.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=81f198a01c0ec25f -C extra-filename=-81f198a01c0ec25f --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern ctor=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libctor-6d15aecbd8ecf9a9.so --extern ghost=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libghost-a7fa8d8cb581322e.so --extern inventory_impl=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libinventory_impl-07295296bec98a10.so --cap-lints allow`
     Compiling indoc v0.3.4
       Running `rustc --edition=2018 --crate-name indoc /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/indoc-0.3.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=3e8e6670af4a3851 -C extra-filename=-3e8e6670af4a3851 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern indoc_impl=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libindoc_impl-b6ef86cd971e397d.so --extern proc_macro_hack=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro_hack-6c301fa525410f51.so --cap-lints allow`
     Compiling paste v0.1.7
       Running `rustc --edition=2018 --crate-name paste /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/paste-0.1.7/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=331a39d39ec573ee -C extra-filename=-331a39d39ec573ee --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern paste_impl=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libpaste_impl-0100b2f59cf859eb.so --extern proc_macro_hack=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro_hack-6c301fa525410f51.so --cap-lints allow`
       Running `rustc --crate-name serde /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.104/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="serde_derive"' --cfg 'feature="std"' -C metadata=ef69b68005b70c62 -C extra-filename=-ef69b68005b70c62 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern serde_derive=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libserde_derive-c97a5ca23329a0e7.so --cap-lints allow --cfg ops_bound --cfg core_reverse --cfg de_boxed_c_str --cfg de_boxed_path --cfg de_rc_dst --cfg core_duration --cfg integer128 --cfg range_inclusive --cfg num_nonzero --cfg core_try_from --cfg num_nonzero_signed --cfg std_atomic64 --cfg std_atomic`
     Compiling serde_json v1.0.48
       Running `rustc --edition=2018 --crate-name serde_json /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_json-1.0.48/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=5007cd62ba6989a5 -C extra-filename=-5007cd62ba6989a5 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern itoa=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libitoa-b9ca519a13df71bf.rmeta --extern ryu=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libryu-81c16096c65f1d25.rmeta --extern serde=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libserde-ef69b68005b70c62.rmeta --cap-lints allow`
     Compiling tokenizers v0.7.0 (/tmp/pip-install-d2wcoxbe/tokenizers/tokenizers-lib)
       Running `rustc --edition=2018 --crate-name tokenizers tokenizers-lib/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=02f2af0a4056c877 -C extra-filename=-02f2af0a4056c877 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern clap=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libclap-4d1679758f5cc3c5.rmeta --extern indicatif=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libindicatif-af81cb79d58cbea4.rmeta --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --extern rand=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librand-102d035e4ca6c699.rmeta --extern rayon=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librayon-35b9846c719edc99.rmeta --extern regex=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libregex-40c5630aef8afe3e.rmeta --extern regex_syntax=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libregex_syntax-feb44197369905d4.rmeta --extern serde=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libserde-ef69b68005b70c62.rmeta --extern serde_json=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libserde_json-5007cd62ba6989a5.rmeta --extern unicode_normalization_alignments=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_normalization_alignments-2f95bcef5d770e35.rmeta --extern unicode_categories=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_categories-3376acf28b9791fe.rmeta`
     Compiling pyo3 v0.8.5
       Running `rustc --edition=2018 --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.8.5/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="extension-module"' --cfg 'feature="python3"' -C metadata=7ff152acc5305eee -C extra-filename=-7ff152acc5305eee --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/pyo3-7ff152acc5305eee -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rlib --extern regex=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libregex-40c5630aef8afe3e.rlib --extern serde=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libserde-ef69b68005b70c62.rlib --extern serde_json=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libserde_json-5007cd62ba6989a5.rlib --extern version_check=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libversion_check-cb4ea0451d56bc0d.rlib --cap-lints allow`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/pyo3-7ff152acc5305eee/build-script-build`
  error: failed to run custom build command for `pyo3 v0.8.5`

  Caused by:
    process didn't exit successfully: `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/pyo3-7ff152acc5305eee/build-script-build` (exit code: 101)
  --- stderr
  thread 'main' panicked at 'Error: pyo3 requires a nightly or dev version of Rust.', /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.8.5/build.rs:542:9
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

  warning: build failed, waiting for other jobs to finish...
  error: build failed
  cargo rustc --lib --manifest-path Cargo.toml --features pyo3/extension-module pyo3/python3 --release --verbose -- --crate-type cdylib
  error: cargo failed with code: 101

  ----------------------------------------
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed directly
```



<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

A successful install of `transformers`.

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: `>2.3.0`
- Platform:
```
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
```

- Python version: `3.7.4`
- PyTorch version (GPU?): `1.4.0`
- Tensorflow version (GPU?): N/A.
- Using GPU in script?: N/A.
- Using distributed or parallel set-up in script?: N/A.


Comments:
Which version of Rust are you on? Looking at the trace [PyO3 ](https://pyo3.rs/v0.9.0-alpha.1/)requires at least 1.37.0-nightly 2019-07-19.

I am on version `1.41.0`

Interestingly, their website says 1.37.x is the requirement, but [on GitHub](https://github.com/PyO3/pyo3#usage) they say you need 1.42.0-nightly 2020-01-21. That's quite a harsh requirement, I think, but nothing you can do about it I suppose. (Except installing an older version of PyO3 from source.) Can you try either of those options and let us know?

I went with the former option (installing a nightly build of rust). Here is what I tried:

1. Install [rustup](https://rustup.rs/)

```
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```

2. Install a [nightly build of rust](https://doc.rust-lang.org/book/appendix-07-nightly-rust.html#rustup-and-the-role-of-rust-nightly)

```
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env
```

Try to update `transformers`:

```
pip install --upgrade transformers
```

No beans. Got the following stacktrace:

```
Building wheels for collected packages: tokenizers
  Building wheel for tokenizers (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/johnmg/t2t/bin/python /home/johnmg/t2t/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmp6fk4hgm1
       cwd: /tmp/pip-install-k2pjj650/tokenizers
  Complete output (224 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib
  creating build/lib/tokenizers
  copying tokenizers/__init__.py -> build/lib/tokenizers
  creating build/lib/tokenizers/models
  copying tokenizers/models/__init__.py -> build/lib/tokenizers/models
  creating build/lib/tokenizers/decoders
  copying tokenizers/decoders/__init__.py -> build/lib/tokenizers/decoders
  creating build/lib/tokenizers/normalizers
  copying tokenizers/normalizers/__init__.py -> build/lib/tokenizers/normalizers
  creating build/lib/tokenizers/pre_tokenizers
  copying tokenizers/pre_tokenizers/__init__.py -> build/lib/tokenizers/pre_tokenizers
  creating build/lib/tokenizers/processors
  copying tokenizers/processors/__init__.py -> build/lib/tokenizers/processors
  creating build/lib/tokenizers/trainers
  copying tokenizers/trainers/__init__.py -> build/lib/tokenizers/trainers
  creating build/lib/tokenizers/implementations
  copying tokenizers/implementations/sentencepiece_bpe.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/char_level_bpe.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/byte_level_bpe.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/bert_wordpiece.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/base_tokenizer.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/__init__.py -> build/lib/tokenizers/implementations
  copying tokenizers/__init__.pyi -> build/lib/tokenizers
  copying tokenizers/models/__init__.pyi -> build/lib/tokenizers/models
  copying tokenizers/decoders/__init__.pyi -> build/lib/tokenizers/decoders
  copying tokenizers/normalizers/__init__.pyi -> build/lib/tokenizers/normalizers
  copying tokenizers/pre_tokenizers/__init__.pyi -> build/lib/tokenizers/pre_tokenizers
  copying tokenizers/processors/__init__.pyi -> build/lib/tokenizers/processors
  copying tokenizers/trainers/__init__.pyi -> build/lib/tokenizers/trainers
  running build_ext
  running build_rust
      Updating crates.io index
  warning: unused manifest key: target.x86_64-apple-darwin.rustflags
     Compiling proc-macro2 v1.0.8
     Compiling unicode-xid v0.2.0
     Compiling syn v1.0.15
     Compiling libc v0.2.67
     Compiling lazy_static v1.4.0
     Compiling autocfg v1.0.0
     Compiling cfg-if v0.1.10
     Compiling semver-parser v0.7.0
     Compiling memchr v2.3.3
     Compiling serde v1.0.104
     Compiling regex-syntax v0.6.14
       Running `rustc --crate-name build_script_build --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-1.0.8/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=7f8009cddc5e6def -C extra-filename=-7f8009cddc5e6def --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/build/proc-macro2-7f8009cddc5e6def -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name unicode_xid /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-xid-0.2.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' -C metadata=c4b64db85789a8a8 -C extra-filename=-c4b64db85789a8a8 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/syn-1.0.15/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="clone-impls"' --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="extra-traits"' --cfg 'feature="full"' --cfg 'feature="parsing"' --cfg 'feature="printing"' --cfg 'feature="proc-macro"' --cfg 'feature="quote"' --cfg 'feature="visit"' -C metadata=df69c996af1dadc1 -C extra-filename=-df69c996af1dadc1 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/build/syn-df69c996af1dadc1 -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.67/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=59dfff2fc32cb87b -C extra-filename=-59dfff2fc32cb87b --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/build/libc-59dfff2fc32cb87b -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name lazy_static /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/lazy_static-1.4.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=daaa2cdb90fc8b44 -C extra-filename=-daaa2cdb90fc8b44 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name autocfg /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/autocfg-1.0.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=be76b16d1dfaa3e8 -C extra-filename=-be76b16d1dfaa3e8 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name cfg_if --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/cfg-if-0.1.10/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=796850ba8a8cedaa -C extra-filename=-796850ba8a8cedaa --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name semver_parser /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/semver-parser-0.7.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=69550f148bc5bb95 -C extra-filename=-69550f148bc5bb95 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
     Compiling maybe-uninit v2.0.0
     Compiling ryu v1.0.2
     Compiling getrandom v0.1.14
     Compiling unicode-width v0.1.7
     Compiling itoa v0.4.5
     Compiling scopeguard v1.1.0
     Compiling ppv-lite86 v0.2.6
     Compiling bitflags v1.2.1
     Compiling rayon-core v1.7.0
     Compiling version_check v0.9.1
     Compiling unindent v0.1.5
     Compiling smallvec v1.2.0
     Compiling either v1.5.3
     Compiling strsim v0.8.0
     Compiling vec_map v0.8.1
     Compiling ansi_term v0.11.0
     Compiling number_prefix v0.3.0
     Compiling unicode_categories v0.1.1
     Compiling spin v0.5.2
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/memchr-2.3.3/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' --cfg 'feature="use_std"' -C metadata=1d902e6b0fc561bf -C extra-filename=-1d902e6b0fc561bf --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/build/memchr-1d902e6b0fc561bf -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name regex_syntax /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/regex-syntax-0.6.14/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="unicode"' --cfg 'feature="unicode-age"' --cfg 'feature="unicode-bool"' --cfg 'feature="unicode-case"' --cfg 'feature="unicode-gencat"' --cfg 'feature="unicode-perl"' --cfg 'feature="unicode-script"' --cfg 'feature="unicode-segment"' -C metadata=49413942df53b636 -C extra-filename=-49413942df53b636 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.104/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="serde_derive"' --cfg 'feature="std"' -C metadata=d826302b09b30fa3 -C extra-filename=-d826302b09b30fa3 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/build/serde-d826302b09b30fa3 -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/maybe-uninit-2.0.0/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=733894adef0cf9fb -C extra-filename=-733894adef0cf9fb --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/build/maybe-uninit-733894adef0cf9fb -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ryu-1.0.2/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=5e3d074139bd55e5 -C extra-filename=-5e3d074139bd55e5 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/build/ryu-5e3d074139bd55e5 -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/getrandom-0.1.14/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="std"' -C metadata=df35e20e514661d3 -C extra-filename=-df35e20e514661d3 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/build/getrandom-df35e20e514661d3 -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name unicode_width /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-width-0.1.7/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' -C metadata=23c2035cef4c6900 -C extra-filename=-23c2035cef4c6900 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name itoa /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/itoa-0.4.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=0670cd2bba7e59c0 -C extra-filename=-0670cd2bba7e59c0 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name scopeguard /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/scopeguard-1.1.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=cc84917dee271887 -C extra-filename=-cc84917dee271887 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name ppv_lite86 --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="simd"' --cfg 'feature="std"' -C metadata=013047a1e1834c1c -C extra-filename=-013047a1e1834c1c --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/bitflags-1.2.1/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' -C metadata=836b697d86bba37f -C extra-filename=-836b697d86bba37f --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/build/bitflags-836b697d86bba37f -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.7.0/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=f57886c0482abf7e -C extra-filename=-f57886c0482abf7e --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/build/rayon-core-f57886c0482abf7e -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name version_check /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/version_check-0.9.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=db8b107c34362735 -C extra-filename=-db8b107c34362735 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name unindent --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/unindent-0.1.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=dc1487787c7c90f6 -C extra-filename=-dc1487787c7c90f6 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name smallvec --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/smallvec-1.2.0/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=d5eba03a866e39a3 -C extra-filename=-d5eba03a866e39a3 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name either /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/either-1.5.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=99303bd779c82d42 -C extra-filename=-99303bd779c82d42 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name strsim /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/strsim-0.8.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=e2648e9eff68e95c -C extra-filename=-e2648e9eff68e95c --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name vec_map /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/vec_map-0.8.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=6138169e8c0f8f54 -C extra-filename=-6138169e8c0f8f54 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name ansi_term /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ansi_term-0.11.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=fa4822742d417eef -C extra-filename=-fa4822742d417eef --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name number_prefix /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/number_prefix-0.3.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=e3de789ab67e6629 -C extra-filename=-e3de789ab67e6629 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name unicode_categories /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode_categories-0.1.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=190c74b7ffce666b -C extra-filename=-190c74b7ffce666b --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name spin /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/spin-0.5.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=bbad36607e408080 -C extra-filename=-bbad36607e408080 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow`
     Compiling semver v0.9.0
       Running `rustc --crate-name semver /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/semver-0.9.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' -C metadata=3dc1d287cff8dfce -C extra-filename=-3dc1d287cff8dfce --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern semver_parser=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libsemver_parser-69550f148bc5bb95.rmeta --cap-lints allow`
     Compiling textwrap v0.11.0
       Running `rustc --crate-name textwrap /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/textwrap-0.11.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=50a2c2e63154ee37 -C extra-filename=-50a2c2e63154ee37 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern unicode_width=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libunicode_width-23c2035cef4c6900.rmeta --cap-lints allow`
     Compiling thread_local v1.0.1
       Running `rustc --crate-name thread_local /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/thread_local-1.0.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=eac48644045cdc04 -C extra-filename=-eac48644045cdc04 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern lazy_static=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/liblazy_static-daaa2cdb90fc8b44.rmeta --cap-lints allow`
     Compiling unicode-normalization-alignments v0.1.12
       Running `rustc --crate-name unicode_normalization_alignments /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-normalization-alignments-0.1.12/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=4bec82dc12738e91 -C extra-filename=-4bec82dc12738e91 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern smallvec=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libsmallvec-d5eba03a866e39a3.rmeta --cap-lints allow`
     Compiling rustc_version v0.2.3
       Running `rustc --crate-name rustc_version /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rustc_version-0.2.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=f9f112a037565bee -C extra-filename=-f9f112a037565bee --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern semver=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libsemver-3dc1d287cff8dfce.rmeta --cap-lints allow`
     Compiling crossbeam-utils v0.7.2
     Compiling crossbeam-epoch v0.8.2
     Compiling num-traits v0.2.11
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-utils-0.7.2/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=82d9fcbae2b0fcfe -C extra-filename=-82d9fcbae2b0fcfe --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/build/crossbeam-utils-82d9fcbae2b0fcfe -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern autocfg=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libautocfg-be76b16d1dfaa3e8.rlib --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=d1169acc937644c9 -C extra-filename=-d1169acc937644c9 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/build/crossbeam-epoch-d1169acc937644c9 -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern autocfg=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libautocfg-be76b16d1dfaa3e8.rlib --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/num-traits-0.2.11/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=5b86f099584d3dae -C extra-filename=-5b86f099584d3dae --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/build/num-traits-5b86f099584d3dae -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern autocfg=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libautocfg-be76b16d1dfaa3e8.rlib --cap-lints allow`
       Running `/tmp/pip-install-k2pjj650/tokenizers/target/release/build/rayon-core-f57886c0482abf7e/build-script-build`
       Running `/tmp/pip-install-k2pjj650/tokenizers/target/release/build/memchr-1d902e6b0fc561bf/build-script-build`
       Running `/tmp/pip-install-k2pjj650/tokenizers/target/release/build/bitflags-836b697d86bba37f/build-script-build`
       Running `rustc --crate-name memchr /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/memchr-2.3.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' --cfg 'feature="use_std"' -C metadata=2eb004acc56bfef6 -C extra-filename=-2eb004acc56bfef6 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow --cfg memchr_runtime_simd --cfg memchr_runtime_sse2 --cfg memchr_runtime_sse42 --cfg memchr_runtime_avx`
       Running `rustc --crate-name bitflags /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/bitflags-1.2.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' -C metadata=561e1eaab5576c8d -C extra-filename=-561e1eaab5576c8d --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow --cfg bitflags_const_fn`
       Running `/tmp/pip-install-k2pjj650/tokenizers/target/release/build/libc-59dfff2fc32cb87b/build-script-build`
       Running `rustc --crate-name libc /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.67/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=1ecf199bab423512 -C extra-filename=-1ecf199bab423512 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow --cfg freebsd11 --cfg libc_priv_mod_use --cfg libc_union --cfg libc_const_size_of --cfg libc_align --cfg libc_core_cvoid --cfg libc_packedN`
       Running `/tmp/pip-install-k2pjj650/tokenizers/target/release/build/getrandom-df35e20e514661d3/build-script-build`
       Running `/tmp/pip-install-k2pjj650/tokenizers/target/release/build/ryu-5e3d074139bd55e5/build-script-build`
       Running `/tmp/pip-install-k2pjj650/tokenizers/target/release/build/syn-df69c996af1dadc1/build-script-build`
       Running `/tmp/pip-install-k2pjj650/tokenizers/target/release/build/serde-d826302b09b30fa3/build-script-build`
       Running `/tmp/pip-install-k2pjj650/tokenizers/target/release/build/maybe-uninit-733894adef0cf9fb/build-script-build`
       Running `/tmp/pip-install-k2pjj650/tokenizers/target/release/build/proc-macro2-7f8009cddc5e6def/build-script-build`
       Running `rustc --crate-name maybe_uninit /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/maybe-uninit-2.0.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=d37c4119dc6540c4 -C extra-filename=-d37c4119dc6540c4 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow --cfg derive_copy --cfg repr_transparent --cfg native_uninit`
       Running `rustc --crate-name ryu /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ryu-1.0.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=c0ae351db5af2a03 -C extra-filename=-c0ae351db5af2a03 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow --cfg integer128 --cfg must_use_return --cfg maybe_uninit`
     Compiling memoffset v0.5.3
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/memoffset-0.5.3/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=3cd08936cb404f8e -C extra-filename=-3cd08936cb404f8e --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/build/memoffset-3cd08936cb404f8e -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern rustc_version=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/librustc_version-f9f112a037565bee.rlib --cap-lints allow`
     Compiling c2-chacha v0.2.3
       Running `rustc --crate-name c2_chacha --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/c2-chacha-0.2.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="simd"' --cfg 'feature="std"' -C metadata=6547bb04c22119fb -C extra-filename=-6547bb04c22119fb --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern ppv_lite86=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libppv_lite86-013047a1e1834c1c.rmeta --cap-lints allow`
       Running `rustc --crate-name proc_macro2 --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-1.0.8/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=e43ba7375fb44eb2 -C extra-filename=-e43ba7375fb44eb2 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern unicode_xid=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libunicode_xid-c4b64db85789a8a8.rmeta --cap-lints allow --cfg use_proc_macro --cfg wrap_proc_macro --cfg proc_macro_span`
       Running `/tmp/pip-install-k2pjj650/tokenizers/target/release/build/crossbeam-utils-82d9fcbae2b0fcfe/build-script-build`
       Running `/tmp/pip-install-k2pjj650/tokenizers/target/release/build/num-traits-5b86f099584d3dae/build-script-build`
     Compiling aho-corasick v0.7.8
       Running `rustc --crate-name aho_corasick /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/aho-corasick-0.7.8/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=1aa04c8e211e8977 -C extra-filename=-1aa04c8e211e8977 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern memchr=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libmemchr-2eb004acc56bfef6.rmeta --cap-lints allow`
       Running `/tmp/pip-install-k2pjj650/tokenizers/target/release/build/crossbeam-epoch-d1169acc937644c9/build-script-build`
       Running `rustc --crate-name num_traits /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/num-traits-0.2.11/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=5895a2370c90e52c -C extra-filename=-5895a2370c90e52c --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow --cfg has_i128`
       Running `/tmp/pip-install-k2pjj650/tokenizers/target/release/build/memoffset-3cd08936cb404f8e/build-script-build`
     Compiling quote v1.0.2
       Running `rustc --crate-name quote --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/quote-1.0.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=2a3c58f3767a45fb -C extra-filename=-2a3c58f3767a45fb --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libproc_macro2-e43ba7375fb44eb2.rmeta --cap-lints allow`
       Running `rustc --crate-name memoffset /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/memoffset-0.5.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=2c7b74dca44a9da4 -C extra-filename=-2c7b74dca44a9da4 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --cap-lints allow --cfg memoffset_maybe_uninit --cfg memoffset_doctests`
       Running `rustc --crate-name crossbeam_utils /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-utils-0.7.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=388986d928bc4f32 -C extra-filename=-388986d928bc4f32 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern cfg_if=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libcfg_if-796850ba8a8cedaa.rmeta --extern lazy_static=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/liblazy_static-daaa2cdb90fc8b44.rmeta --cap-lints allow --cfg has_min_const_fn --cfg has_atomic_u8 --cfg has_atomic_u16 --cfg has_atomic_u32 --cfg has_atomic_u64`
       Running `rustc --crate-name syn --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/syn-1.0.15/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="clone-impls"' --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="extra-traits"' --cfg 'feature="full"' --cfg 'feature="parsing"' --cfg 'feature="printing"' --cfg 'feature="proc-macro"' --cfg 'feature="quote"' --cfg 'feature="visit"' -C metadata=f20a4e9749d3ee5d -C extra-filename=-f20a4e9749d3ee5d --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libproc_macro2-e43ba7375fb44eb2.rmeta --extern quote=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libquote-2a3c58f3767a45fb.rmeta --extern unicode_xid=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libunicode_xid-c4b64db85789a8a8.rmeta --cap-lints allow`
     Compiling clicolors-control v1.0.1
     Compiling num_cpus v1.12.0
     Compiling termios v0.3.1
     Compiling atty v0.2.14
       Running `rustc --crate-name getrandom --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/getrandom-0.1.14/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="std"' -C metadata=048e32c5a0c04df6 -C extra-filename=-048e32c5a0c04df6 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern cfg_if=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libcfg_if-796850ba8a8cedaa.rmeta --extern libc=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/liblibc-1ecf199bab423512.rmeta --cap-lints allow`
       Running `rustc --crate-name clicolors_control /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/clicolors-control-1.0.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="terminal_autoconfig"' -C metadata=0b0b6007b4183ec9 -C extra-filename=-0b0b6007b4183ec9 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern lazy_static=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/liblazy_static-daaa2cdb90fc8b44.rmeta --extern libc=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/liblibc-1ecf199bab423512.rmeta --cap-lints allow`
       Running `rustc --crate-name num_cpus /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/num_cpus-1.12.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=d96ba1da53092c10 -C extra-filename=-d96ba1da53092c10 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern libc=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/liblibc-1ecf199bab423512.rmeta --cap-lints allow`
       Running `rustc --crate-name termios /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/termios-0.3.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=ec3a50c8d1bc7e85 -C extra-filename=-ec3a50c8d1bc7e85 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern libc=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/liblibc-1ecf199bab423512.rmeta --cap-lints allow`
       Running `rustc --crate-name atty /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/atty-0.2.14/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=a326ad27cb30e935 -C extra-filename=-a326ad27cb30e935 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern libc=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/liblibc-1ecf199bab423512.rmeta --cap-lints allow`
     Compiling crossbeam-queue v0.2.1
       Running `rustc --crate-name crossbeam_epoch /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=91bb1210b79b03da -C extra-filename=-91bb1210b79b03da --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern cfg_if=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libcfg_if-796850ba8a8cedaa.rmeta --extern crossbeam_utils=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libcrossbeam_utils-388986d928bc4f32.rmeta --extern lazy_static=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/liblazy_static-daaa2cdb90fc8b44.rmeta --extern maybe_uninit=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libmaybe_uninit-d37c4119dc6540c4.rmeta --extern memoffset=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libmemoffset-2c7b74dca44a9da4.rmeta --extern scopeguard=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libscopeguard-cc84917dee271887.rmeta --cap-lints allow --cfg has_min_const_fn`
       Running `rustc --crate-name crossbeam_queue /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-queue-0.2.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=c776341a9b185621 -C extra-filename=-c776341a9b185621 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern cfg_if=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libcfg_if-796850ba8a8cedaa.rmeta --extern crossbeam_utils=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libcrossbeam_utils-388986d928bc4f32.rmeta --cap-lints allow`
     Compiling clap v2.33.0
       Running `rustc --crate-name clap /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-2.33.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="ansi_term"' --cfg 'feature="atty"' --cfg 'feature="color"' --cfg 'feature="default"' --cfg 'feature="strsim"' --cfg 'feature="suggestions"' --cfg 'feature="vec_map"' -C metadata=3071ac3e668ed07c -C extra-filename=-3071ac3e668ed07c --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern ansi_term=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libansi_term-fa4822742d417eef.rmeta --extern atty=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libatty-a326ad27cb30e935.rmeta --extern bitflags=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libbitflags-561e1eaab5576c8d.rmeta --extern strsim=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libstrsim-e2648e9eff68e95c.rmeta --extern textwrap=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libtextwrap-50a2c2e63154ee37.rmeta --extern unicode_width=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libunicode_width-23c2035cef4c6900.rmeta --extern vec_map=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libvec_map-6138169e8c0f8f54.rmeta --cap-lints allow`
     Compiling rand_core v0.5.1
       Running `rustc --crate-name rand_core --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rand_core-0.5.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="alloc"' --cfg 'feature="getrandom"' --cfg 'feature="std"' -C metadata=7ce6542ec4257257 -C extra-filename=-7ce6542ec4257257 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern getrandom=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libgetrandom-048e32c5a0c04df6.rmeta --cap-lints allow`
     Compiling regex v1.3.4
       Running `rustc --crate-name regex /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/regex-1.3.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="aho-corasick"' --cfg 'feature="default"' --cfg 'feature="memchr"' --cfg 'feature="perf"' --cfg 'feature="perf-cache"' --cfg 'feature="perf-dfa"' --cfg 'feature="perf-inline"' --cfg 'feature="perf-literal"' --cfg 'feature="std"' --cfg 'feature="thread_local"' --cfg 'feature="unicode"' --cfg 'feature="unicode-age"' --cfg 'feature="unicode-bool"' --cfg 'feature="unicode-case"' --cfg 'feature="unicode-gencat"' --cfg 'feature="unicode-perl"' --cfg 'feature="unicode-script"' --cfg 'feature="unicode-segment"' -C metadata=88e281e069c7ba33 -C extra-filename=-88e281e069c7ba33 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern aho_corasick=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libaho_corasick-1aa04c8e211e8977.rmeta --extern memchr=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libmemchr-2eb004acc56bfef6.rmeta --extern regex_syntax=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libregex_syntax-49413942df53b636.rmeta --extern thread_local=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libthread_local-eac48644045cdc04.rmeta --cap-lints allow`
     Compiling crossbeam-deque v0.7.3
       Running `rustc --crate-name crossbeam_deque /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-deque-0.7.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=0c2beaa054f6da4d -C extra-filename=-0c2beaa054f6da4d --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern crossbeam_epoch=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libcrossbeam_epoch-91bb1210b79b03da.rmeta --extern crossbeam_utils=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libcrossbeam_utils-388986d928bc4f32.rmeta --extern maybe_uninit=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libmaybe_uninit-d37c4119dc6540c4.rmeta --cap-lints allow`
     Compiling rand_chacha v0.2.1
       Running `rustc --crate-name rand_chacha --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rand_chacha-0.2.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="std"' -C metadata=9c3e099a39a894e1 -C extra-filename=-9c3e099a39a894e1 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern c2_chacha=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libc2_chacha-6547bb04c22119fb.rmeta --extern rand_core=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/librand_core-7ce6542ec4257257.rmeta --cap-lints allow`
       Running `rustc --crate-name rayon_core --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.7.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=43e482f55cc760b6 -C extra-filename=-43e482f55cc760b6 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern crossbeam_deque=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libcrossbeam_deque-0c2beaa054f6da4d.rmeta --extern crossbeam_queue=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libcrossbeam_queue-c776341a9b185621.rmeta --extern crossbeam_utils=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libcrossbeam_utils-388986d928bc4f32.rmeta --extern lazy_static=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/liblazy_static-daaa2cdb90fc8b44.rmeta --extern num_cpus=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libnum_cpus-d96ba1da53092c10.rmeta --cap-lints allow`
     Compiling rand v0.7.3
       Running `rustc --crate-name rand --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rand-0.7.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="alloc"' --cfg 'feature="default"' --cfg 'feature="getrandom"' --cfg 'feature="getrandom_package"' --cfg 'feature="libc"' --cfg 'feature="std"' -C metadata=d905e01484b0667a -C extra-filename=-d905e01484b0667a --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern getrandom_package=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libgetrandom-048e32c5a0c04df6.rmeta --extern libc=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/liblibc-1ecf199bab423512.rmeta --extern rand_chacha=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/librand_chacha-9c3e099a39a894e1.rmeta --extern rand_core=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/librand_core-7ce6542ec4257257.rmeta --cap-lints allow`
     Compiling rayon v1.3.0
       Running `rustc --crate-name rayon --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.3.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=5f29404b086ee91b -C extra-filename=-5f29404b086ee91b --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern crossbeam_deque=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libcrossbeam_deque-0c2beaa054f6da4d.rmeta --extern either=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libeither-99303bd779c82d42.rmeta --extern rayon_core=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/librayon_core-43e482f55cc760b6.rmeta --cap-lints allow`
     Compiling console v0.9.2
       Running `rustc --crate-name console --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/console-0.9.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="unicode-width"' -C metadata=2ce4edc7b4c8449e -C extra-filename=-2ce4edc7b4c8449e --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern clicolors_control=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libclicolors_control-0b0b6007b4183ec9.rmeta --extern lazy_static=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/liblazy_static-daaa2cdb90fc8b44.rmeta --extern libc=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/liblibc-1ecf199bab423512.rmeta --extern regex=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libregex-88e281e069c7ba33.rmeta --extern termios=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libtermios-ec3a50c8d1bc7e85.rmeta --extern unicode_width=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libunicode_width-23c2035cef4c6900.rmeta --cap-lints allow`
     Compiling indicatif v0.14.0
       Running `rustc --crate-name indicatif --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/indicatif-0.14.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' -C metadata=e47d5e054f023436 -C extra-filename=-e47d5e054f023436 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern console=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libconsole-2ce4edc7b4c8449e.rmeta --extern lazy_static=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/liblazy_static-daaa2cdb90fc8b44.rmeta --extern number_prefix=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libnumber_prefix-e3de789ab67e6629.rmeta --extern regex=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libregex-88e281e069c7ba33.rmeta --cap-lints allow`
     Compiling pyo3-derive-backend v0.8.5
       Running `rustc --crate-name pyo3_derive_backend --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-derive-backend-0.8.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=aba0e2d131928acb -C extra-filename=-aba0e2d131928acb --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libproc_macro2-e43ba7375fb44eb2.rmeta --extern quote=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libquote-2a3c58f3767a45fb.rmeta --extern syn=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libsyn-f20a4e9749d3ee5d.rmeta --cap-lints allow`
     Compiling serde_derive v1.0.104
     Compiling proc-macro-hack v0.5.11
     Compiling ghost v0.1.1
     Compiling ctor v0.1.13
     Compiling inventory-impl v0.1.5
     Compiling pyo3cls v0.8.5
       Running `rustc --crate-name serde_derive /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_derive-1.0.104/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 --cfg 'feature="default"' -C metadata=034c70940b2eedef -C extra-filename=-034c70940b2eedef --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libproc_macro2-e43ba7375fb44eb2.rlib --extern quote=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libquote-2a3c58f3767a45fb.rlib --extern syn=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libsyn-f20a4e9749d3ee5d.rlib --extern proc_macro --cap-lints allow`
       Running `rustc --crate-name ghost --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ghost-0.1.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=883c66593baf1317 -C extra-filename=-883c66593baf1317 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libproc_macro2-e43ba7375fb44eb2.rlib --extern quote=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libquote-2a3c58f3767a45fb.rlib --extern syn=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libsyn-f20a4e9749d3ee5d.rlib --extern proc_macro --cap-lints allow`
       Running `rustc --crate-name proc_macro_hack --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro-hack-0.5.11/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=3cf880fc746cfd7d -C extra-filename=-3cf880fc746cfd7d --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libproc_macro2-e43ba7375fb44eb2.rlib --extern quote=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libquote-2a3c58f3767a45fb.rlib --extern syn=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libsyn-f20a4e9749d3ee5d.rlib --extern proc_macro --cap-lints allow`
       Running `rustc --crate-name ctor --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ctor-0.1.13/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=2c98d245a0bd2934 -C extra-filename=-2c98d245a0bd2934 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern quote=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libquote-2a3c58f3767a45fb.rlib --extern syn=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libsyn-f20a4e9749d3ee5d.rlib --extern proc_macro --cap-lints allow`
       Running `rustc --crate-name inventory_impl --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/inventory-impl-0.1.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=753bfdf5c2c1b356 -C extra-filename=-753bfdf5c2c1b356 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libproc_macro2-e43ba7375fb44eb2.rlib --extern quote=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libquote-2a3c58f3767a45fb.rlib --extern syn=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libsyn-f20a4e9749d3ee5d.rlib --extern proc_macro --cap-lints allow`
       Running `rustc --crate-name pyo3cls --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3cls-0.8.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=3b5ed485ca9e08fe -C extra-filename=-3b5ed485ca9e08fe --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libproc_macro2-e43ba7375fb44eb2.rlib --extern pyo3_derive_backend=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libpyo3_derive_backend-aba0e2d131928acb.rlib --extern quote=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libquote-2a3c58f3767a45fb.rlib --extern syn=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libsyn-f20a4e9749d3ee5d.rlib --extern proc_macro --cap-lints allow`
     Compiling paste-impl v0.1.7
     Compiling indoc-impl v0.3.4
       Running `rustc --crate-name paste_impl --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/paste-impl-0.1.7/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=84e729460d896f1b -C extra-filename=-84e729460d896f1b --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern proc_macro_hack=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libproc_macro_hack-3cf880fc746cfd7d.so --extern proc_macro2=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libproc_macro2-e43ba7375fb44eb2.rlib --extern quote=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libquote-2a3c58f3767a45fb.rlib --extern syn=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libsyn-f20a4e9749d3ee5d.rlib --extern proc_macro --cap-lints allow`
       Running `rustc --crate-name indoc_impl --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/indoc-impl-0.3.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=3a756aee825cada7 -C extra-filename=-3a756aee825cada7 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern proc_macro_hack=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libproc_macro_hack-3cf880fc746cfd7d.so --extern proc_macro2=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libproc_macro2-e43ba7375fb44eb2.rlib --extern quote=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libquote-2a3c58f3767a45fb.rlib --extern syn=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libsyn-f20a4e9749d3ee5d.rlib --extern unindent=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libunindent-dc1487787c7c90f6.rlib --extern proc_macro --cap-lints allow`
  error: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libproc_macro_hack-3cf880fc746cfd7d.so)
    --> /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/indoc-impl-0.3.4/src/lib.rs:12:5
     |
  12 | use proc_macro_hack::proc_macro_hack;
     |     ^^^^^^^^^^^^^^^

  error: aborting due to previous error

  error: could not compile `indoc-impl`.

  Caused by:
    process didn't exit successfully: `rustc --crate-name indoc_impl --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/indoc-impl-0.3.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=3a756aee825cada7 -C extra-filename=-3a756aee825cada7 --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern proc_macro_hack=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libproc_macro_hack-3cf880fc746cfd7d.so --extern proc_macro2=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libproc_macro2-e43ba7375fb44eb2.rlib --extern quote=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libquote-2a3c58f3767a45fb.rlib --extern syn=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libsyn-f20a4e9749d3ee5d.rlib --extern unindent=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libunindent-dc1487787c7c90f6.rlib --extern proc_macro --cap-lints allow` (exit code: 1)
  warning: build failed, waiting for other jobs to finish...
  error: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libproc_macro_hack-3cf880fc746cfd7d.so)
   --> /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/paste-impl-0.1.7/src/lib.rs:6:5
    |
  6 | use proc_macro_hack::proc_macro_hack;
    |     ^^^^^^^^^^^^^^^

  error: aborting due to previous error

  error: could not compile `paste-impl`.

  Caused by:
    process didn't exit successfully: `rustc --crate-name paste_impl --edition=2018 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/paste-impl-0.1.7/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=84e729460d896f1b -C extra-filename=-84e729460d896f1b --out-dir /tmp/pip-install-k2pjj650/tokenizers/target/release/deps -L dependency=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps --extern proc_macro_hack=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libproc_macro_hack-3cf880fc746cfd7d.so --extern proc_macro2=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libproc_macro2-e43ba7375fb44eb2.rlib --extern quote=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libquote-2a3c58f3767a45fb.rlib --extern syn=/tmp/pip-install-k2pjj650/tokenizers/target/release/deps/libsyn-f20a4e9749d3ee5d.rlib --extern proc_macro --cap-lints allow` (exit code: 1)
  warning: build failed, waiting for other jobs to finish...
  error: build failed
  cargo rustc --lib --manifest-path Cargo.toml --features pyo3/python3 pyo3/extension-module --release --verbose -- --crate-type cdylib
  error: cargo failed with code: 101

  ----------------------------------------
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed directly
```

I am not sure if this is a version issue or an incompatibility issue, cf. and related: https://stackoverflow.com/questions/55363823/redhat-centos-glibc-2-18-not-found

Because of cross-platform compatibility, it might better to have `tokenizers` as an optional dependency and set `use_fast` to False by default instead to True.

Hi @JohnGiorgi,  

Thanks for reporting this issue and sorry you're having trouble with transformers. 

As @BramVanroy mentioned pyo3 package (which allows us to build the Python binding to Rust) requires some features only available in nightly release. However 1.42 is only required for the version `9.0-alpha` of pyo3 which we're not currently using.

I did try to reproduce the error you linked above on a Fedora machine I've at hand and wasn't able to. Can you provide some more information regarding this machine ? On which platform is it running ? x86_64 ? POWER9 ? Also if you can include some `uname -a ` output that would be very helpful.

Any information you might be able to provide to us will help us track down this build issue.

Many thanks 

Looking at the trace, it seems `tokenizers` uses v0.8.5 of PyO3, which (according to their docs) requires 1.37.0-nightly 2019-07-19. So it's a bit odd that installation didn't work for OP on 1.41 from the start. But perhaps it has to do with GLIBC_2.18?

@mfuntowicz From the CentOS fora, [it seems that 2.18 will never come to CentOS 7](https://forums.centos.org/viewtopic.php?t=71740) so I fear this is just an incompatibility. That adds to my point that I would suggest that `tokenizers` is an optional dependency and use_fast should be False by default. 


@BramVanroy The nightly features can take many release cycles before landing on a stable version. So these features are probably still part of the nightly only, even in the stable `1.4.1`.

Anyway, we provide `manylinux` wheels, and these are built on `CentOS 5` so I think the real problem here is to find out why it didn't download the wheel in the first place, but tried to re-compile instead.

@n1t0 Ah, that makes sense. Thanks for the clarification.

@JohnGiorgi Can you try to install `tokenizers` with `pip debug install tokenizers -vv`? It'll show you all compatible tags.

@mfuntowicz

Output of `lscpu`

```
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                40
On-line CPU(s) list:   0-39
Thread(s) per core:    1
Core(s) per socket:    20
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping:              4
CPU MHz:               2057.373
CPU max MHz:           3700.0000
CPU min MHz:           1000.0000
BogoMIPS:              4800.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              28160K
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 invpcid_single intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke md_clear spec_ctrl intel_stibp flush_l1d
```

Output of `uname -a`

```
Linux beluga1.int.ets1.calculquebec.ca 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 x86_64 GNU/Linux
```

Output of `cat /etc/os-release`

```
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
```

I am working on a Compute Canada cluster, so information about it can also be found [here](https://docs.computecanada.ca/wiki/B%C3%A9luga/en).

@BramVanroy 

Here is the output of `pip debug install tokenizers -vv`

```
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
pip version: pip 20.0.2 from /home/johnmg/t2t/lib/python3.7/site-packages/pip (python 3.7)
sys.version: 3.7.4 (default, Jul 18 2019, 19:34:02)
[GCC 5.4.0]
sys.executable: /home/johnmg/t2t/bin/python
sys.getdefaultencoding: utf-8
sys.getfilesystemencoding: utf-8
locale.getpreferredencoding: UTF-8
sys.platform: linux
sys.implementation:
  name: cpython
'cert' config value: install, wheel, :env:
REQUESTS_CA_BUNDLE: None
CURL_CA_BUNDLE: /etc/pki/tls/certs/ca-bundle.crt
pip._vendor.certifi.where(): /home/johnmg/t2t/lib/python3.7/site-packages/pip/_vendor/certifi/cacert.pem
Compatible tags: 27
  cp37-cp37m-linux_x86_64
  cp37-abi3-linux_x86_64
  cp37-none-linux_x86_64
  cp36-abi3-linux_x86_64
  cp35-abi3-linux_x86_64
  cp34-abi3-linux_x86_64
  cp33-abi3-linux_x86_64
  cp32-abi3-linux_x86_64
  py37-none-linux_x86_64
  py3-none-linux_x86_64
  py36-none-linux_x86_64
  py35-none-linux_x86_64
  py34-none-linux_x86_64
  py33-none-linux_x86_64
  py32-none-linux_x86_64
  py31-none-linux_x86_64
  py30-none-linux_x86_64
  cp37-none-any
  py37-none-any
  py3-none-any
  py36-none-any
  py35-none-any
  py34-none-any
  py33-none-any
  py32-none-any
  py31-none-any
  py30-none-any
```

I'm also having these same errors. It gets past the install, but then when running the tests `pip install -e ".[testing]"` I get: 

`error: Can not find Rust compiler
  ----------------------------------------
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed directly`

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hey guys, @JohnGiorgi @Shane-Neeley , I think I figured out what was happening to you (to me also). At the time of your question, I think there was a high possibility that in the "setup.py" of transformers' source code, there was a line writing "tokenizers=0.5.2" ; but in fact this old version will bother the update of transformers.
But when I install the newest tokenizers by pip (pip install -U tokenizers), you will get a tokenizers==0.7.0. That's why at the time of your question, there were always conflits and bugs about this tokenizers (I also installed Rust and setuptool-rust, it was always the same error).

And now they just corrected this line in the setup.py. So I just suggest you to 
1. uninstall your old version transformers (very important!)
2. pip install -U tokenizers (so that it becomes tokenizers==0.7.0)
3. install transformers from source !

Then voilà ! You'll get a brand new smoothy transformers.

> Hey guys, @JohnGiorgi @Shane-Neeley , I think I figured out what was happening to you (to me also). At the time of your question, I think there was a high possibility that in the "setup.py" of transformers' source code, there was a line writing "tokenizers=0.5.2" ; but in fact this old version will bother the update of transformers.
> But when I install the newest tokenizers by pip (pip install -U tokenizers), you will get a tokenizers==0.7.0. That's why at the time of your question, there were always conflits and bugs about this tokenizers (I also installed Rust and setuptool-rust, it was always the same error).
> 
> And now they just corrected this line in the setup.py. So I just suggest you to
> 
> 1. uninstall your old version transformers (very important!)
> 2. pip install -U tokenizers (so that it becomes tokenizers==0.7.0)
> 3. install transformers from source !
> 
> Then voilà ! You'll get a brand new smoothy transformers.

I am installing transformers from source but I am getting the error.

`curl https://sh.rustup.rs -sSf | sh`
`source $HOME/.cargo/env`
`pip3 install --upgrade transformers`

these lines worked for me

> `curl https://sh.rustup.rs -sSf | sh` `source $HOME/.cargo/env` `pip3 install --upgrade transformers`
> 
> эти строки сработали для меня

Подскажите, пожалуйста, куда именно Вы вводите данные команды?

Please tell me where exactly you enter these commands?