add json2pickledset to remove duplicate entries and pickle for further processing