java procedure of de-duplicate for flume, first unfinished version.