apache/spark — pull request #2142 by davies (2014-08-26 18:51:42+00:00)
Title: [SPARK-2871] [PySpark] add countApproxDistinct() API

RDD.countApproxDistinct(relativeSD=0.05):

:: Experimental ::
Return approximate number of distinct elements in the RDD.

The algorithm used is based on streamlib's implementation of
"HyperLogLog in Practice: Algorithmic Engineering of a State
of The Art Cardinality Estimation Algorithm", available
here.

This support all the types of objects, which is supported by
Pyrolite, nearly all builtin types.

@param relativeSD Relative accuracy. Smaller values create
counters that require more space.
It must be greater than 0.000017.

>>> n = sc.parallelize(range(1000)).map(str).countApproxDistinct()
>>> 950 < n < 1050
True