Addition of polite crawler