Package: datasketches
The Apache DataSketches Library for PythonThis is the official version of theApache DataSketchesPython library.In the analysis of big data there are often problem queries that don’t scale because they require huge compute resources and time to generate exact results. Examples include count distinct, quantiles, most-frequent items, joins, matrix computations, and graph analysis.If approximate results are acceptable, there is a class of specialized algorithms, called streaming algorithms, or sketches that can produce results orders-of magnitude faster and with mathematically proven error bounds. For interactive queries there may not be other viable alternatives, and in the case of real-time analysis, sketches are the only known solution.This package provides a variety of sketches as described below. Wherever a specific type of sketch exists in Apache DataSketches packages for other languages, the sketches will be portable between languages (for platforms with the same endianness).Building and InstallationOnce cloned, the library can be installed by runningpython3 -m pip install .in the project root directory which will also install the necessary dependencies, namely NumPy andNanobind.If you prefer to call thesetup.pybuild script directly, which is discouraged, you must first installnanobindandnumpy, as well as any other dependencies listed under the build-system section inpyproject.toml.The library is also available from PyPI viapython3 -m pip install datasketches.UsageHaving installed the library, loading the Apache DataSketches Library in Python is simple:import datasketches.The unit tests are mostly structured in a tutorial style and can be used as a reference example for how to feed data into and query the different types of sketches.Available Sketch ClassesKLL (Absolute Error Quantiles)kll_ints_sketchkll_floats_sketchkll_doubles_sketchkll_items_sketchQuantiles (Absolute Error Quantiles, inferior algorithm)quantiles_ints_sketchquantiles_floats_sketchquantiles_doubles_sketchquantiles_items_sketchREQ (Relative Error Quantiles)req_ints_sketchreq_floats_sketchreq_items_sketchFrequent Itemsfrequent_strings_sketchfrequent_items_sketchError types arefrequent_items_error_type.{NO_FALSE_NEGATIVES | NO_FALSE_POSITIVES}Thetaupdate_theta_sketchcompact_theta_sketch(cannot be instantiated directly)theta_uniontheta_intersectiontheta_a_not_btheta_jaccard_similarityTupleupdate_tuple_sketchcompact_tuple_sketch(cannot be instantiated directly)tuple_uniontuple_intersectiontuple_a_not_btuple_jaccard_similarityHLLhll_sketchhll_unionTarget HLL types aretgt_hll_type.{HLL_4 | HLL_6 | HLL_8}CPCcpc_sketchcpc_unionVarOpt Samplingvar_opt_sketchvar_opt_unionEBPPS Sampling (Exactly proportional to weight)ebpps_sketchVector of KLLvector_of_kll_ints_sketchesvector_of_kll_floats_sketchesKolmogorov-Smirnov Testks_testapplied to a pair of matched-type Absolute Error quantiles sketchesDensitydensity_sketchCount-min sketchcount_min_sketchKnown Differences from C++The Python API largely mirrors the C++ API, with a few minor exceptions: The primary known differences are that Python on modern platforms does not support unsigned integer values or numeric values with fewer than 64 bits. As a result, you may not be able to produce identical sketches from within Python as you can with Java and C++. Loading those sketches after they have been serialized from another language will work as expected.The Vector of KLL object is currently exclusive to python, and holds an array of independent KLL sketches. This is useful for creating a set of KLL sketches over a vector and has been designed to allow input as either a vector or a matrix of multiple vectors.We have also removed reliance on a builder class for theta sketches as Python allows named arguments to the constructor, not strictly positional arguments.Developer InstructionsThe only developer-specific instructions relate to running unit tests.Unit testsThe Python unit tests are run viatox, with no arguments, from the project root directory. Tox creates a temporary virtual environment in which to build and run the unit tests. In the event you are missing the necessary package, tox may be installed withpython3 -m pip install --upgrade tox.LicenseThe Apache DataSketches Library is distributed under the Apache 2.0 License.There may be precompiled binaries provided as a convenience and distributed through PyPI via [https://pypi.org/project/datasketches/] contain compiled code fromnanobind, which is distributed under a BSD license.