flattening and removing duplicates