corrected check for duplicates fetching all records and use array_diff