updated algorithm for duplicate checking - needs to be moved to backend