updated to remove duplicate documents