Repository: huggingface/tokenizers (language: rust)

Issue:
Update outdated dependencies and feature-gate CLI
Making the library a bit more up to date with common dependencies.
Unsure whether special testing is required for the cli.

Added a feature-gate to cli but kept it as default. This allows someone only using the library who doesn't wish to use the binary at all to opt out using `default-features = false`

Also put dirs behind the feature gate http, as it's only used hogether with cached_path in the from_pretrained-flow


Discussion / hints:
Hi Thanks for this PR.

Do you mind splitting the feature gates from the dependencies update in separate PRs ?

It's much easier for me to add the cli-feature gate as I understand the implications more easily.

For dependency updates, we need to run a full chain of tests (in `transformers` to make sure nothing is broken there). 
This typically is done before any release too, but for dependency updates I would also do it before merging on master as it's easier then to figure out what dependency might be breaking things.
> Hi Thanks for this PR.
> 
> Do you mind splitting the feature gates from the dependencies update in separate PRs ?
> 
> It's much easier for me to add the cli-feature gate as I understand the implications more easily.
> 
> For dependency updates, we need to run a full chain of tests (in `transformers` to make sure nothing is broken there). This typically is done before any release too, but for dependency updates I would also do it before merging on master as it's easier then to figure out what dependency might be breaking things.

No problem, I did leave the dirs feature gating in since it's only used in a single place in the codebase, which is already behind an existing feature gate. That PR is here https://github.com/huggingface/tokenizers/pull/960
Thanks you for changing the PR !

Fix patch:
diff --git a/tokenizers/Cargo.toml b/tokenizers/Cargo.toml
index 2d90a326a..b6fdc3ac3 100644
--- a/tokenizers/Cargo.toml
+++ b/tokenizers/Cargo.toml
@@ -47,7 +47,7 @@ lazy_static = "1.4"
 rand = "0.8"
 onig = { version = "6.4", default-features = false, optional = true }
 regex = "1.9"
-regex-syntax = "0.7"
+regex-syntax = "0.8"
 rayon = "1.8"
 rayon-cond = "0.3"
 serde = { version = "1.0", features = [ "derive" ] }
@@ -57,7 +57,7 @@ unicode-normalization-alignments = "0.1"
 unicode_categories = "0.1"
 unicode-segmentation = "1.10"
 indicatif = {version = "0.17", optional = true}
-itertools = "0.11"
+itertools = "0.12"
 log = "0.4"
 derive_builder = "0.12"
 spm_precompiled = "0.1"
@@ -66,7 +66,7 @@ aho-corasick = "1.1"
 paste = "1.0.14"
 macro_rules_attribute = "0.2.0"
 thiserror = "1.0.49"
-fancy-regex = { version = "0.11", optional = true}
+fancy-regex = { version = "0.13", optional = true}
 getrandom = { version = "0.2.10" }
 esaxx-rs = { version = "0.1.10", default-features = false, features=[]}
 monostate = "0.1.9"


Test patch: