allow indexing for robots