Package: json-stream-rs-tokenizer
NOTE:json-stream-rs-tokenizeris now automatically used byjson-stream, so unless you find a bug, you can ignore this package's
existence!json-stream-rs-tokenizerA faster tokenizer for thejson-streamPython library.It's actually justjson-stream's own tokenizer (itself adapted from theNAYAproject) ported to Rust almost
verbatim and made available as a Python module usingPyO3.On my machine, itspeeds up parsing by a factor of 4–10, depending on the
nature of the data.InstallationImplicitStarting at its 2.0 release,json-streamdepends on and usesjson-stream-rs-tokenizerby default, so you don't need to install it
explicitly anymore.ExplicitIf you use an olderjson-streamversion (which you have no reason to do) or
need to installjson-stream-rs-tokenizerexplicitly for another reason, you
can do:pipinstalljson-stream-rs-tokenizerThe library will be installed as a prebuilt wheel if one is available for your
platform. Otherwise, pip will try to build it from the source distribution,
which requires a Rust toolchain to be installed and available to succeed.Note that if the build from source fails, the package installation will be
considered successfully completed anyway, butRustTokenizer(see below) won't
be available for import. This is so that packages (specifically,json-stream)
can depend on the library but fall back to their own implementation if neither
a prebuilt wheel is available nor the build succeeds.You can increase the installation command's verbosity with-v(repeated for
even more information, e.g.-vv) to see error messages when the build from
source fails.Notethat if the Rust library is compiled in debug mode, it will runslowerthan the pure-Python tokenizer. The setuptools configuration should
make sure this doesn't happen even when installing in development mode, but
when in doubt, run installation commands with-vto see the Rust compilation
commands and verify that they used--release.UsageImplicitAs described above,json-stream-rs-tokenizeris now used byjson-streamby
default, so you don't have to do anything special to use it.json-streamwill
fall back to its pure-Python tokenizer whenjson-stream-rs-tokenizerwas not
successfully installed, however.ExplicitFor older versions ofjson-stream, or if you want toensurethe Rust
tokenizer is used no matter what, simply pass this package'sRustTokenizeras
thetokenizerargument tojson-stream'sloadorvisit:fromioimportStringIOfromjson_streamimportloadfromjson_stream_rs_tokenizerimportRustTokenizerjson_buf=StringIO('{ "a": [1,2,3,4], "b": [5,6,7] }')# uses the Rust tokenizer to load JSON:d=load(json_buf,tokenizer=RustTokenizer)fork,lind.items():print(f"{k}:{' '.join(str(n)forninl)}")Note that the import ofRustTokenizerwill fail if the Rust extension is not
available (i.e., when no prebuilt wheels were available and the installation
from the source distribution failed).LimitationsFor PyPy, the speedup is only 1.0-1.5x (much lower than that for CPython).
This has yet to beinvestigated.In builds that don't support PyO3'snum-bigintextension(currently only PyPy builds and manual ones against Python's limited C API
(Py_LIMITED_API)), conversion of large integers is performed in Python
rather than in Rust, at a very small runtime cost.BenchmarksThe package comes with a script for rudimentary benchmarks on randomly
generated JSON data. To run it, you'll need to install the optionalbenchmarkdependencies:pipinstall'json-stream-rs-tokenizer[benchmark]'You can then run the benchmark as follows:python-mjson_stream_rs_tokenizer.benchmarkRun it with--helpto see more information.TestsTo run the tests, you'll need to install the optionaltestdependencies:pipinstall'json-stream-rs-tokenizer[test]'As the test dependencies depend on the benchmark dependencies but the feature
enabling such"recursive optional dependencies"was only introduced in Pip 21.3, you'll need a version of Pip at least as
recent as that. For older versions, just install the test dependencies
manually.LicenseMIT license. Refer to theLICENSEfile for details.