Package: hyperjson
A hyper-fast, safe Python module to read and write JSON data. Works as a
drop-in replacement for Python's built-injsonmodule.
This is alpha software and there will be bugs, so maybe don't deploy to productionjustyet. :wink:Installationpip install hyperjsonUsagehyperjson is meant as a drop-in replacement for Python'sjson
module:>>>importhyperjson>>>hyperjson.dumps([{"key":"value"},81,True])'[{"key":"value"},81,true]'>>>hyperjson.loads("""[{"key": "value"}, 81, true]""")[{u'key':u'value'},81,True]MotivationParsing JSON is a solved problem; so, no need to reinvent the wheel, right?Well, unless you care aboutperformance and safety.Turns out, parsing JSONcorrectlyis ahard problem. Thanks to Rust however, we can minimize the risk of running intostack overflows or segmentation faultshowever.hyperjson is a thin wrapper around Rust'sserde-jsonandpyo3. It is compatible with Python 3 (and 2 on a best-effort basis).For a more in-depth discussion,watch the talk about this project recorded at the Rust Cologne Meetup in August 2018.GoalsCompatibility: Support the full feature-set of Python'sjsonmodule.Safety: No segfaults, panics, or overflows.Performance: Significantly faster thanjsonand as fast asujson(both written in C).Non-goalsSupport ujson and simplejson extensions:Custom extensions likeencode(),__json__(), ortoDict()are not
supported. The reason is, that they go against PEP8 (e.g.dundermethods
are restricted to the standard library, camelCase is not Pythonic) and are not
available in Python'sjsonmodule.Whitespace preservation: Whitespace in JSON strings is not preserved.
Mainly because JSON is a whitespace-agnostic format andserde-jsonstrips
them out by default. In practice this should not be a problem, since your
application must not depend on whitespace padding, but it's something to be
aware of.BenchmarkWe arenotfast yet. That said, we haven't done any big optimizations.
In the long-term we might explore features of newer CPUs like multi-core and SIMD.
That's one area other (C-based) JSON extensions haven't touched yet, because it might
make code harder to debug and prone to race-conditions. In Rust, this is feasible due to crates likefasterorrayon.So there's a chance that the following measurements might improve soon.If you want to help, check the instructions in theDevelopment Environmentsection below.Test machine:MacBook Pro 15 inch, Mid 2015 (2,2 GHz Intel Core i7, 16 GB RAM) Darwin 17.6.18Contributions welcome!If you would like to hack on hyperjson, here's what needs to be done:Implementloads()Implementload()Implementdumps()Implementdump()Benchmark againstjsonandujson(see#1)Add a CI/CD pipeline for easier testing (see#2)Create a proper pip package from it, to make installing easier (see#3).Profile and optimize performance (see#16)Add remainingkeyword-only argumentsto methodsJust pick one of the open tickets. We can provide mentorship if you like. :smiley:Developer guideThis project usespipenvfor managing the development environment. If you don't have it installed, runpip install poetryThe project requires thenightlyversion of Rust.Install it viarustup:rustup install nightlyIf you have already installed thenightlyversion, make sure it is up-to-date:rustup update nightlyAfter that, you can compile the current version of hyperjson and execute all tests and benchmarks with the following commands:make install
make test
make bench🤫 Pssst!... runmake helpto learn more.Drawing pretty diagramsIn order to recreate the benchmark histograms, you first need a few additional prerequisites:MatplotlibNumpyOn macOS, please also add the following to your~/.matplotlib/matplotlibrc(reference):backend: TkAggAfter that, run the following:make plotLicensehyperjson is licensed under either ofApache License, Version 2.0, (LICENSE-APACHE orhttp://www.apache.org/licenses/LICENSE-2.0)MIT license (LICENSE-MIT orhttp://opensource.org/licenses/MIT)at your option.ContributionUnless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in hyperjson by you, as defined in the Apache-2.0 license, shall
be dual licensed as above, without any additional terms or conditions.