apache/incubator-spark — pull request #73 by falaki (2013-10-18 05:46:20+00:00)
Title: Approximate distinct count

Added countDistinct() to RDD and countDistinctByKey() to PairRDDFunctions to approximately count distinct number of elements and distinct number of values per key, respectively. Both functions use HyperLogLog from stream-lib for counting. Both functions take a parameter that controls the trade-off between accuracy and memory consumption. Also added Scala docs and test suites for both methods.