refactored removed_duplicates and added this to various pages, removed needless null checking