Package: ddsketch
ddsketchThis repo contains the Python implementation of the distributed quantile sketch
algorithm DDSketch [1]. DDSketch has relative-error guarantees for any quantile
q in [0, 1]. That is if the true value of the qth-quantile isxthen DDSketch
returns a valueysuch that|x-y| / x < ewhereeis the relative error
parameter. (The default here is set to 0.01.)  DDSketch is also fully mergeable,
meaning that multiple sketches from distributed systems can be combined in a
central node.Our default implementation,DDSketch, is guaranteed [1] to not grow too large
in size for any data that can be described by a distribution whose tails are
sub-exponential.We also provide implementations (LogCollapsingLowestDenseDDSketchandLogCollapsingHighestDenseDDSketch) where the q-quantile will be accurate up to
the specified relative error for q that is not too small (or large). Concretely,
the q-quantile will be accurate up to the specified relative error as long as it
belongs to one of thembins kept by the sketch.  If the data is time in
seconds, the default ofm = 2048covers 80 microseconds to 1 year.InstallationTo install this package, runpip install ddsketch, or clone the repo and runpython setup.py install. This package depends onnumpyandprotobuf. (The
protobuf dependency can be removed if it's not applicable.)Usagefrom ddsketch import DDSketch

sketch = DDSketch()Add values to the sketchimport numpy as np

values = np.random.normal(size=500)
for v in values:
  sketch.add(v)Find the quantiles ofvaluesto within the relative error.quantiles = [sketch.get_quantile_value(q) for q in [0.5, 0.75, 0.9, 1]]Merge anotherDDSketchintosketch.another_sketch = DDSketch()
other_values = np.random.normal(size=500)
for v in other_values:
  another_sketch.add(v)
sketch.merge(another_sketch)The quantiles ofvaluesconcatenated withother_valuesare still accurate to within the relative error.DevelopmentTo work on ddsketch a Python interpreter must be installed. It is recommended to use the provided development
container (requiresdocker) which includes all the required Python interpreters.docker-compose run devOr, if developing outside of docker then it is recommended to use a virtual environment:pip install virtualenv
virtualenv --python=3 .venv
source .venv/bin/activateTestingTo run the tests installriot:pip install riotReplace the Python version with the interpreter(s) available.# Run tests with Python 3.9
riot run -p3.9 testRelease notesNew features, bug fixes, deprecations and other breaking changes must have
release notes included.To generate a release note for the change:riot run reno new <short-description-of-change-no-spaces>Edit the generated file to include notes on the changes made in the commit/PR
and add commit it.FormattingFormat code withriot run fmtType-checkingType checking is done withmypy:riot run mypyType-checkingLint the code withflake8:riot run flake8ProtobufThe protobuf is stored in the go repository:https://github.com/DataDog/sketches-go/blob/master/ddsketch/pb/ddsketch.protoInstall the minimum required protoc and generate the Python code:dockerrun-v$PWD:/code-itubuntu:18.04/bin/bash
aptupdate&&aptinstallprotobuf-compiler# default is 3.0.0protoc--proto_path=ddsketch/pb/--python_out=ddsketch/pb/ddsketch/pb/ddsketch.protoReleasingGenerate the release notes and usepandocto format
them for Github:gitcheckoutmaster&&gitpullriotrun-srenoreport--no-show-source|pandoc-frst-tgfm--wrap=noneCopy the output into a new release:https://github.com/DataDog/sketches-py/releases/new.Enter a tag for the release (followingsemver) (eg.v1.1.3,v1.0.3,v1.2.0).Use the tag without thevas the title.Save the release as a draft and pass the link to someone else to give a quick review.If all looks good hit publishReferences[1] Charles Masson and Jee E Rim and Homin K. Lee. DDSketch: A fast and fully-mergeable quantile sketch with relative-error guarantees. PVLDB, 12(12): 2195-2205, 2019. (The code referenced in the paper, including our implementation of the the Greenwald-Khanna (GK) algorithm, can be found at:https://github.com/DataDog/sketches-py/releases/tag/v0.1)