FIX: check ignore robots for urlkey