Politeness

Crawler must obey robots.txt. User can still override this setting.