Detect (and remove) duplications in large cluster. Using Hadoop and HBase.