update spider to handle inconsistencies