adjust robots url