Processed new data and cleaned duplicates